Im using Logastash to collecting data from mysql. Json result looks :
"_source" : {
"username" : "room_test#localhost",
"timestamp" : 1481785195811703,
"peer" : "user#localhost/1596084304715518942270426",
"bare_peer" : "user#localhost",
"xml" : "<message to='room_test1481784717020#localhost' type='groupchat' from='user#localhost'><body>msg</body><jid>456-345</jid></message>",
"txt" : "msg",
"id" : 6452,
"kind" : "groupchat",
"nick" : "user",
"created_at" : "2016-12-15T06:59:55.000Z",
"#version" : "1",
"#timestamp" : "2017-02-25T12:17:52.043Z"
}
I need extract value from "xml" key as separate value. Like this "jid": 456-345
Thank you
Grok can handle this.
grok {
match => {
"xml" => "<jid>(?<jid>[-0-9]{4,9})</jid>"
}
}
This will create a jid field equal to the value you supplied. The value in the JID tag in the XML can be between 4 and 9 characters long. Adjust as you need to.
Related
I am totally new to groovy script and would like some help to solve this out. I have a JSON response I want to manipulate and get desired parameters back by avoiding duplication. The Json response does not have indexes like 0,1,2.. that I can iterate through.
Here is the response that I want to work with:
{
"AuthenticateV2" : {
"displayName" : "Verification of authentication",
"description" : "notification ",
"smsTemplate" : "authentication.v2.0_sms",
"emailHeaderTemplate" : "v2.0_header",
"emailBodyTemplate" : "html",
"parameters" : {
"displayName" : "USER_DISPLAY_NAME",
"actionTokenURL" : "VERIFICATION_LINK",
"customToken" : "VERIFICATION_CODE"
},
"supportedPlans" : [
"connectGo"
]
},
"PasswordRecovery" : {
"displayName" : "Verification of password recovery",
"description" : "notification",
"smsTemplate" : "recovery.v1.0_sms",
"emailHeaderTemplate" : "recovery.v1.0_header",
"emailBodyTemplate" : "recovery.v1.0_body_html",
"parameters" : {
"displayName" : "USER_DISPLAY_NAME",
"actionTokenURL" : "VERIFICATION_LINK",
"customToken" : "VERIFICATION_CODE",
"adminInitiated" : false,
"authnId" : "AUTHENTICATION_IDENTIFIER",
"authnType" : "EMAIL",
"user" : {
"displayName" : "USER_DISPLAY_NAME"
}
},
"supportedPlans" : [
"connectGo"
]
},
"PasswordReset" : {
"displayName" : "password reset",
"description" : "notification",
"smsTemplate" : "recovery.v1.0_sms",
"emailHeaderTemplate" : "recovery.v1.0_header",
"emailBodyTemplate" : "html",
"parameters" : {
"displayName" : "USER_DISPLAY_NAME",
"user" : {
"displayName" : "USER_DISPLAY_NAME"
}
}
The expected output that I want to have:
{
"displayName" : "USER_DISPLAY_NAME",
"actionTokenURL" : "VERIFICATION_LINK",
"customToken" : "VERIFICATION_CODE",
"customToken" : "VERIFICATION_CODE",
"adminInitiated" : false,
"authnId" : "AUTHENTICATION_IDENTIFIER",
"authnType" : "EMAIL"
}
I need to retrieve all fields under parameters tag and also want to avoid duplication
You should first get familiar with parsing and producing JSON in Groovy.
Then, assuming the provided response is a valid JSON (it's not - there are 2 closing curlies (}) missing at the end) to get all the parameters keys merged into one JSON we have to convert the JSON string into a Map object first using JsonSlurper:
def validJsonResponse = '<your valid JSON string>'
Map parsedResponse = new JsonSlurper().parseText(validJsonResponse) as Map
Now, when we have a parsedResponse map we can iterate over all the root items in the response and transform them into the desired form (which is all the unique parameters keys) using Map::collectEntries method:
Map uniqueParameters = parsedResponse.collectEntries { it.value['parameters'] }
Finally, we can convert the uniqueParameters result back into a pretty printed JSON string using JsonOuput:
println JsonOutput.prettyPrint(JsonOutput.toJson(uniqueParameters))
After applying all the above we'll get the output
{
"displayName": "USER_DISPLAY_NAME",
"actionTokenURL": "VERIFICATION_LINK",
"customToken": "VERIFICATION_CODE",
"adminInitiated": false,
"authnId": "AUTHENTICATION_IDENTIFIER",
"authnType": "EMAIL",
"user": {
"displayName": "USER_DISPLAY_NAME"
}
}
If you want to get rid of user entry from the final output just remove it from the resulting uniqueParameters map (uniqueParameters.remove('user')) before converting it back to JSON string.
I am generating random data from the following JSON/AVRO schema:
{
"type" : "record",
"namespace" : "test",
"name" : "metro_data",
"fields": [
{
"name" : "PersonID",
"type" : "int"
},
{
"name" : "TripStartStation",
"type" : {
"type" : "enum",
"name" : "StartStation",
"symbols" : ["WIEHLE_RESTON_EAST", "SPRING_HILL", "GREENSBORO"]
}
},
{
"name" : "TripEndStation",
"type" : {
"type" : "enum",
"name" : "EndStation",
"symbols" : ["WIEHLE_RESTON_EAST", "SPRING_HILL", "GREENSBORO""]
}
}
]
}
The above schema generates this, for example:
[ {
"PersonID" : -1089196095,
"TripStartStation" : "WIEHLE_RESTON_EAST",
"TripEndStation" : "SPRING_HILL"
}
I want to take the PersonID number of the schema, and add it to the Attributes. Eg, the blank in this photo needs to pull the actual PersonID number generated from the flow:
I have tried to use EvaluateJSONPath with the following configuration, and that's how I end up with the empty string set under PersonalID:
Is my next processor UpdateAttribute? Not sure how to pull that content. Thanks!
You are having array of json message(s)(ex: [...]) and You need to split the array of json into individual flowfiles using SplitJson processor with split expression as $.*
Then use EvaluateJsonProcessor to extract PersonID value as a attribute.
Flow:
--> SplitJson --> EvaluateJsonPath--> other processors
For more details refer to this link regards to the same issue.
How can I push new key and value in JSON array?
I tried I used push keyword in update query but I got a different output. I used:
db.users.updateOne({"name":"viki"},{$push{"address.district":"thambaram"}})
I have this document:
{ "_id" : ObjectId("58934f10c7592b1494fd9a4d"), "name" : "viki", "age" : 100, "subject" : [ "c", "node.js", "java" ], "address" : { "city" : "chennai", "state" : "tamilnadu", "pincode" : "123" } }
I want to add "district":"thambaram" in address json array
I need like:
{ "_id" : ObjectId("58934f10c7592b1494fd9a4d"), "name" : "viki", "age" : 100, "subject" : [ "c", "node.js", "java" ], "address" : { "city" : "chennai", "state" : "tamilnadu", "pincode" : "123","district":"thambaram"} }
Use $set
db.users.updateOne({"name":"viki"},{$set:{"address.district":"thambaram"}})
This should work.
The $push operator appends a specified value to an array. In your case you should use $set
I have a MongoDB looking like this:
[
{
"status" : 0,
"name" : "Yaknow",
"email" : "yaknow#not.this",
"_id" : "5875a42ea469f40c684de385"
},
{
"status" : 1,
"name" : "johnk",
"email" : "johnk#not#this",
"_id" : "586e31c6ce07af6f891f80fd"
}
]
Meanwhile, all the emails have changed and I got a Json with the new ones:
[
{
"email" : "yaknow#gmai.new",
"_id" : "5875a42ea469f40c684de385"
},
{
"email" : "johnk#gmail.new",
"_id" : "586e31c6ce07af6f891f80fd"
}
]
How do I update all the emails?
There is no operator in mongodb which allows you modify string value by replacing some part of string. You should get documents, and then for each of documents you should locally prepare updated value and update document:
db.collection.find({}).forEach(function(doc){
var newEmail = doc.email.substr(0, doc.email.indexOf('#')) + "#gmail.new";
db.collection.update({_id: doc._id}, {$set:{email: newEmail}});
});
I have a Spring Roo project I am trying to create based on log4mongo-java appender and I want to get access to the data entries that looks like:
{
"_id" : ObjectId("4f16cd30b138685057c8ebcb"),
"timestamp" : ISODate("2012-01-18T13:46:24.704Z"),
"level" : "INFO", "thread" : "catalina-exec-8180-3",
"message" : "method execution[execution(TerminationComponent.terminateCall(..))]",
"loggerName" :
{ "fullyQualifiedClassName" : "component_logger",
"package" : ["component_logger"],
"className" : "component_logger"
},
"properties" : {
"cookieId" : "EDE44DC03EB65D91657885A34C80595E"
},
"fileName" : "LoggingAspect.java",
"method" : "logForComponent",
"lineNumber" : "81", "class" : {
"fullyQualifiedClassName" : "com.comcast.ivr.core.aspects.LoggingAspect",
"package" : ["com", "comcast", "ivr", "core", "aspects", "LoggingAspect"],
"className" : "LoggingAspect"
},
"host" : {
"process" : "2220#pacdcivrqaapp01",
"name" : "pacdcivrqaapp01",
"ip" : "24.40.31.85"
},
"applicationName" : "D2",
"eventType" : "Development"
}
The timestamp looks like:
"timestamp" : ISODate("2012-01-17T22:30:19.839Z")
How can I add a field in my Logging domain object to map this field?
That's just the JavaScript Date (according to the mongo docs, and as can be demonstrated in the shell), so try with java.util.Date.