I have the json with unix-timestamp field. I like to extract year from it.
So, for example:
{"eventno": "event1",
"unixtimestamp": 1589379890}
Expected result:
{"eventno": "event1",
"unixtime": 2020}
I try to do this using JoltTransfromJSON and NiFi expression language, but my attempts failed. One of them:
[
{
"operation": "shift",
"spec": {
"unixtime": "${unixtimestamp:multiply(1000):format('yyyy', 'GMT')}"
}
}
]
How can I transform it?
#GrigorySkvortsov
The Expression Language syntax should be:
${attribute:expressionLanguage():functions()}
If what you have above isn't just a typo retest after removing the } after unixtimestamp.
Unit Test outside of Jolt Transform with updateAttribute Processor to dial in the correct Expression Language chain. Here is an example I made to test it:
Then the 4 values are:
Related
I'm having issues with getting an epoch conversion to Timestamp to work properly. So far my example timestamp looks like the following:
{"createTime": 1510932843000}
What my end goal is to make it look like the following:
2017-11-17 3:34:03.000
The things I've tried so far are the UpdateRecord and JoltTransformation Processor. For the UpdateRecord I have tried various ways but all end in an error. The current code I have for this is:
${field.value:format("yyyy-MM-dd HH:mm:ss.SSS")}
Which results in the following error:
JSON Object due to java.lang.NumberFormatException: For input string: "2017-11-17 15:34:03.000": For input string: "2017-11-17 15:34:03.000"
I have also tried the code without the multiply(1000) to the same effect.
I have also tried a Jolt Transformation of the following code:
{
"createTime": "${createTime:append('000'):format('yyyy-MM-dd HH:mm:ss.SSS')}"
}
This however results in the following:
"createTime": "1970-01-01 00:00:00.000"
Which isn't what I'm looking for as its the incorrect date result. Am I doing something wrong within my code itself or is another factor occurring? I've been working with this and searching all over for different kind of results and have tried multiple different formats with no success. Any help with this would be greatly appreciated!
My preferred solution:
Use a ScriptedTransformRecord processor:
Record Reader: JsonTreeReader
Record Writer: JsonRecordSetWriter
Script Language: Groovy
Script Body:
import java.time.Instant;
import java.time.format.DateTimeFormatter;
import java.time.ZoneId;
def formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.SSS").withZone(ZoneId.of("Europe/Bratislava"))
record.setValue("createTimeFormatted", formatter.format(Instant.ofEpochMilli(record.getAsLong("createTime"))))
return record
Output json:
{
"createTime" : 1510932843000,
"createTimeFormatted" : "2017-11-17 16:34:03.000"
}
Another approach
Use 2 processors: JoltTransformJSON (convert type from Long to String) -> UpdateRecord (convert date).
JoltTransformJSON processor:
Jolt Transformation DSL: Chain
Jolt specification:
[
{
"operation": "modify-overwrite-beta",
"spec": {
"createTime": "=toString"
}
}
]
UpdateRecord processor:
Record Reader: JsonTreeReader
Record Writer: JsonRecordSetWriter
Replacement Value Strategy: Literal Value
/createTime (dynamic property): ${field.value:format("yyyy-MM-dd HH:mm:ss.SSS")}
Output json:
{
"createTime" : "2017-11-17 16:34:03.000"
}
I am new to shell scripting and I need some help.
I am trying to use jq to get values from a api response and check for its correctness.
Here is a sample for how the response looks like,
{
"data" : {
"transactionType" : "Sales",
"transactionSubType" : "DomesticSale",
"Items" : [ {
"itemID" : "2",
"itemType" : "Good",
"amount" : 5.0,
"tax" : 1.0
} ]
}
}
I am able to get the values for transactionType or transactionsubtype or even ItemID values etc as given below
jq '.data.transactionType'
jq '.data.Items[0].itemID'
for Transaction type and item id
but when it comes to values of numeric types i.e., without the quotes in it, I don't get any value.
I am using similar syntax for the numeric type also as shown below.
jq '.data.Items[0].amount'
jq '.data.Items[0].tax'
Please help!!!
Your jq invocations are fine, but the sample data is missing a final closing brace ("}"), so perhaps you were not feeding jq properly.
If you're wondering why you didn't see an error message, it's almost certainly because jq 1.5 is not very good about handling incomplete JSON. The problem has since been fixed at "master". With the current version, you'd see something like this:
parse error: Unfinished JSON term at EOF at line 15, column 0
I want to remove all fields from a json except the one named foo. I used transformation spec as given below:
[
{
"operation": "remove",
"spec": {
"^(?!foo).*$": ""
}
}
]
I tried executing this on http://jolt-demo.appspot.com/#inception but it doesn't work and it outputs the input json, untransformed. Am I doing something wrong?
Yeah so, "shift" does to support any "regex" matching other than "", so "^(?!foo).$" is not going to work.
I think you are better off, using "shift" to match "foo" and copy it across to the output. Anyting not matched by the "shift" spec does not get copied across to the output.
Spec
[
{
"operation": "shift",
"spec": {
// matches top level key "foo" in the intput, and copies the
// value at that location to the output map with key "foo".
"foo" : "foo"
}
}
]
Shift copies data from the input to a new ouput, all the other operations (default, remove, cardinality, etc) modify the input.
I have a JSON response like below
{
"queryStartDate": "20170523134739822",
"queryEndDate": "20170623134739822",
"Rows": [
{
"hasScdHistoryOnly": false,
"Values": [
"1",
"53265",
"CO"
]
},
{
"hasScdHistoryOnly": false,
"Values": [
"1",
"137382",
"CO"
]
},
{
"hasScdHistoryOnly": false,
"Values": [
"1",
"310824",
"CO"
]
}
]
}
I am using Jmeter's JSON Extractor post-processor to receive the second value from the last of the 'Values' list. i.e. 53265, 137382, 310824.
I've tried to use $.Rows[*].Values[-2:-1], and $.Rows[*].Values[(#.length-2)], according to Stefan's introduction: http://goessner.net/articles/JsonPath/index.html#e2, but neither of them are working. Would you please help me out?
I believe JMeter is using JayWay JSON Path library, so you should be looking for the documentation here instead.
In general I would recommend using JSR223 PostProcessor as an alternative to JSON Path Extractors, both are applicable for basic scenarios only, when it comes to advanced queries and operators their behaviour is flaky.
Add JSR223 PostProcessor as a child of the request which returns above JSON
Make sure you have "groovy" selected in the "Language" drop down and "Cache compiled script if available" box is ticked
Put the following code into "Script" area
def values = com.jayway.jsonpath.JsonPath.parse(prev.getResponseDataAsString()).read('$..Values')
values.eachWithIndex { val, idx ->
vars.put('yourVar_' + (idx + 1), val.get(val.size()-2))
}
It should generate the following JMeter Variables:
yourVar_1=53265
yourVar_2=137382
yourVar_3=310824
which seem to be something you're looking for.
References:
Groovy: Parsing and producing JSON
Apache Groovy - Why and How You Should Use It
Using View Results tree's JSon Path Tester I could see that the following expression you used for extracting the values were not correct (correct for online JSONPath Online Evaluator but not working for JMeter)
Used Expression: $.Rows[*].Values[-2:-1]
Output from JSon Path Tester: No Match Found.
Used Expression: $.Rows[*].Values[(#.length-2)]
Output from JSon Path Tester: Exception: Could not parse token starting at position 16. Expected ?, ', 0-9, *
If the expression $.Rows[*].Values[1] is used it extracts the desired responses.
Used Expression: $.Rows[*].Values[1]
Output from JSon Path Tester:
Result[0]=53265
Result[1]=137382
Result[2]=310824
You know how XSLT and other XML processing languages support the "null transformation" which passes a document through unmodified ?
I would like to do the same thing for Jolt (a very nice JSON transformation library used in Apache Camel and other places).
I could use JOLT's "insert default" feature and stick some harmless JSON tag and value at the top level of the document.. which is almost what want. But I couldnt' figure out how to pass through the document through JOLT but leave it untouched.
Why do i want to do this you ask ? We are developing a streaming data pipeline and I have to validate incoming strings as valid JSON... Jolt does that for me for free, but in some cases I don't want to monkey with the document. So, I want to use JOLT as a step in the pipeline, but (in some cases) have it do nothing to the input JSSON doc.
Another option is to create a custom transformer.
package com.example;
public class NullTransform implements Transform{
#Override
public Object transform(Object input) {
return input;
}
}
then reference it from the chainr jolt as below
[
{
"operation": "com.example.NullTransform"
}
]
You'll still incur the deserializaion/serialization overhead but no other code is run.
OOTB Jolt contains 5 "operations" that can be applied to the input hydrated Json. 4 of those (default, remove, sort, cardinality) are mutation operations / modify the supplied hydrated Json. I you gave those 4 an empty "spec", they would do nothing, and your data would "pass thru".
The "shift" operation does not mutate the input it is given. Instead it "copies" data from the "input" to a new "output" map/list. If you don't give "shift" a spec, then it copies nothing across.
Thus, from you question, it sounds like you are talking about "shift". With shift you have to explicitly pass thru all the things you "want to keep".
Depending on your data this may be terrible or easy, as you can have shift copy very large chunks of data across.
Example, for the "inception" example on the jolt demo site. http://jolt-demo.appspot.com/#inception
This Spec basically passes thru the input, coping the whole nested map that is "rating" thru to the output.
[
{
"operation": "shift",
"spec": {
"rating": "rating"
}
}
]
It can be generalized with wildcards to :
Spec
[
{
"operation": "shift",
"spec": {
"*": "&"
}
}
]