I have this:
{
"service" : {
"category" : "managed-object",
"resource" : "attribute",
"action" : "delete",
"options" : {
"uuid" : "#VALUE",
"attributes" : {
"name" : {
"value" : "#VALUE"
},
"contactInfo" : "",
"activationDate" : "",
"deactivationDate" : "",
"protectStopDate" : "",
"processStartDate" : ""
}
}
}
}
I need this:
{
"service" : {
"category" : "managed-object",
"resource" : "attribute",
"action" : "delete",
"options" : {
"uuid" : "#VALUE",
"attributes" : {
"name" : {
"value" : "#VALUE"
}
}
}
}
}
Previously I had a similar problem that was a little bit more complicated, and I received this amazingly simple answer from someone here:
.service.options |= (del(.max, .objectGroupMember) | .attributes|={name})
That jq command works even here, but of course (.max, .objectGroupMember) does not make sense because it does not exist.
How can I achieve my desired result?
A different take on the problem:
walk(if type=="object" then with_entries(select(.value!="")) else . end)
I found it in Weeble's answer to my earlier question:
'.service.options.attributes |= {name}'
Related
I would like to map the following structure
{
"id" : "OUTER_ID",
"name" : "OUTER_NAME"
"items" : [
{
"id" : "INNER_ID_1",
"name" : "INNER_NAME_1",
},
{
"id" : "INNER_ID_2",
"name" : "INNER_NAME_2",
}
]
}
into this
{
"payload": [
{
"key" : "INNER_NAME_1_KEY",
"data" : {
"id" : "OUTER_ID",
"name" : "OUTER_NAME",
"items" : [
{
"id" : "INNER_ID_1",
"name" : "INNER_NAME_1"
}
]
}
},
{
"key" : "INNER_NAME_2_KEY",
"data" : {
"id" : "OUTER_ID",
"name" : "OUTER_NAME",
"items" : [
{
"id" : "INNER_ID_2",
"name" : "INNER_NAME_2"
}
]
}
}
]
}
So, for each item in the initial items array, I want to create an entry in the output's payload, i.e I want to map items[i] to payload[i].data.items while also creating the payload, key and data keys in the output, and setting payload[i].data.id and payload[i].data.name to the input's outer id and name.
Can this be done with jq?
Sure, you can use the following filter :
.id as $id | .name as $name | {payload : [ .items[] | {key:.id, data:{id:$id, name: $name, items:[.]}} ] }
You can try it here.
I have elastic(2.4.6) my_index/_settings:
{
"my_index" : {
"settings" : {
"index" : {
"number_of_shards" : "5",
"preserve_existing" : "true",
"max_result_window" : "10000000",
"{index" : {
"max_result_window" : ""
},
"creation_date" : "66666666",
"number_of_replicas" : "0",
"uuid" : "test",
"version" : {
"created" : "666"
}
}
}
}
}
Can I remove this part:
"{index" : {
"max_result_window" : ""
},
1. How can I do it?
2. Does this row affect on elastic?
No you can not delete this line. but the empty value is OK and This means that it returns to the default value (10000) . in your case I think it set to 100000.
I am trying to find a value in the json file and based on that I need to get the entire json data instead of that particular block.
Here is my sample json
[{
"name" : "Redirect to Website 1",
"behaviors" : [ {
"name" : "redirect",
"options" : {
"mobileDefaultChoice" : "DEFAULT",
"destinationProtocol" : "HTTPS",
"destinationHostname" : "SAME_AS_REQUEST",
"responseCode" : 302
}
} ],
"criteria" : [ {
"name" : "requestProtocol",
"options" : {
"value" : "HTTP"
}
} ],
"criteriaMustSatisfy" : "all"
},
{
"name" : "Redirect to Website 2",
"behaviors" : [ {
"name" : "redirect",
"options" : {
"mobileDefaultChoice" : "DEFAULT",
"destinationProtocol" : "HTTPS",
"destinationHostname" : "SAME_AS_REQUEST",
"responseCode" : 301
}
} ],
"criteria" : [ {
"name" : "contentType",
"options" : {
"matchOperator" : "IS_ONE_OF",
"values" : [ "text/html*", "text/css*", "application/x-javascript*" ],
}
} ],
"criteriaMustSatisfy" : "all"
}]
I am trying to match for "name" : "redirect" inside each behaviors array and if it matches then I need the entire block including the "criteria" section, as you can see its under same block {}
I managed to find the values using select methods but not able to get the parent section.
https://jqplay.org/s/BWJwVdO3Zv
Any help is much appreciated!
To avoid unwanted duplication:
.[]
| first(select(.behaviors[].name == "redirect"))
Equivalently:
.[]
| select(any(.behaviors[]; .name == "redirect"))
You can try this jq command:
<file jq 'select(.[].behaviors[].name=="redirect")'
I'm trying to determine the best way to calculate the elapsed time it took for each operation, series of actions. Looking at my example data below, how might I take the min/max for the "actions" array, for each corresponding operation, which includes 'take' and 'throw' actions:
{
"name" : "test",
"location" : "here",
"operation" "hammer use",
"actions" : [
{
"action" : "take",
"object" : "hammer",
"timestamp" : "12332234234"
},
{
"action" : "drop",
"object" : "hammer",
"timestamp" : "12332234255"
},
{
"action" : "take",
"object" : "hammer",
"timestamp" : "12332234266"
},
{
"action" : "throw",
"object" : "hammer",
"timestamp" : "12332234277"
}
},
{
"name" : "test 2",
"location" : "there",
"operation" : "rock use",
"actions" : [
{
"action" : "take",
"object" : "rock",
"timestamp" : "12332534277"
},
{
"action" : "drop",
"object" : "rock",
"timestamp" : "12332534288"
},
{
"action" : "take",
"object" : "rock",
"timestamp" : "12332534299"
},
{
"action" : "throw",
"object" : "rock",
"timestamp" : "12332534400"
},
{
"name" : "test 3",
"location" : "elsewhere",
"operation" : "seal hose",
"actions" : [
{
"action" : "create",
"object" : "grommet",
"timestamp" : "12332534277"
},
{
"action" : "place",
"object" : "grommet",
"timestamp" : "12332534288"
},
{
"action" : "tighten",
"object" : "hose",
"timestamp" : "12332534299"
}
}
Expected output:
{
"operation" : "hammer use",
"elapsed_time" : 123
},
{
"operation" : "rock use",
"elapsed_time" : 123
}
I'm still new to rethinkdb and trying to get a hang for it. So far, I've come up with the following query to pick the specific records, i'm interested in, from the table:
r.db('test').table('operations').filter(function(row) {
return row('actions').contains(function(x) {
return x('action').eq('take')}).and(
row('actions').contains(function(x) { return x('action').eq('throw') })
);
});
I'm still trying to figure out how to aggregate the results by taking the min/max of the timestamp and subtracting them from each other.
I hope there's enough detail there to get an idea for the goal at hand. Let me know otherwise. Any help greatly appreciated.
Well, nobody tugged on this so I had to solve it without any help. Took a bit longer but finally figured out how. Here's the pseudocode for finding min/max on the nested fields above, and elapsed_time:
r.db('test').table('operations').filter(function(row) {
return row('actions').contains(function(x) { return x('action').eq("take") }).and(
row('actions').contains(function(x) { return x('action').eq("throw") })
);
}).map(function(doc) {
return {
operation: doc('operation'),
min: doc('actions')('timestamp').min(),
max: doc('actions')('timestamp').max(),
elapsed_time: doc('actions')('timestamp').max().sub(doc('actions')('timestamp').min())
}
})
I'm trying to convert JSON into Avro using the kite-sdk morphline module. After playing around I'm able to convert the JSON into Avro using a simple schema (no complex data types).
Then I took it one step further and modified the Avro schema as displayed below (subrec.avsc). As you can see the schema consist of a subrecord.
As soon as I tried to convert the JSON to Avro using the morphlines.conf and the subrec.avsc it failed.
Somehow the JSON paths "/record_type[]/alert/action" are not translated by the toAvro function.
The morphlines.conf
morphlines : [
{
id : morphline1
importCommands : ["org.kitesdk.**"]
commands : [
# Read the JSON blob
{ readJson: {} }
{ logError { format : "record: {}", args : ["#{}"] } }
# Extract JSON
{ extractJsonPaths { flatten: false, paths: {
"/record_type[]/alert/action" : /alert/action,
"/record_type[]/alert/signature_id" : /alert/signature_id,
"/record_type[]/alert/signature" : /alert/signature,
"/record_type[]/alert/category" : /alert/category,
"/record_type[]/alert/severity" : /alert/severity
} } }
{ logError { format : "EXTRACTED THIS : {}", args : ["#{}"] } }
{ extractJsonPaths { flatten: false, paths: {
timestamp : /timestamp,
event_type : /event_type,
source_ip : /src_ip,
source_port : /src_port,
destination_ip : /dest_ip,
destination_port : /dest_port,
protocol : /proto,
} } }
# Create Avro according to schema
{ logError { format : "WE GO TO AVRO"} }
{ toAvro { schemaFile : /etc/flume/conf/conf.empty/subrec.avsc } }
# Create Avro container
{ logError { format : "WE GO TO BINARY"} }
{ writeAvroToByteArray { format: containerlessBinary } }
{ logError { format : "DONE!!!"} }
]
}
]
And the subrec.avsc
{
"type" : "record",
"name" : "Event",
"fields" : [ {
"name" : "timestamp",
"type" : "string"
}, {
"name" : "event_type",
"type" : "string"
}, {
"name" : "source_ip",
"type" : "string"
}, {
"name" : "source_port",
"type" : "int"
}, {
"name" : "destination_ip",
"type" : "string"
}, {
"name" : "destination_port",
"type" : "int"
}, {
"name" : "protocol",
"type" : "string"
}, {
"name": "record_type",
"type" : ["null", {
"name" : "alert",
"type" : "record",
"fields" : [ {
"name" : "action",
"type" : "string"
}, {
"name" : "signature_id",
"type" : "int"
}, {
"name" : "signature",
"type" : "string"
}, {
"name" : "category",
"type" : "string"
}, {
"name" : "severity",
"type" : "int"
}
] } ]
} ]
}
The output on { logError { format : "EXTRACTED THIS : {}", args : ["#{}"] } } I output the following:
[{
/record_type[]/alert / action = [allowed],
/record_type[]/alert / category = [],
/record_type[]/alert / severity = [3],
/record_type[]/alert / signature = [GeoIP from NL,
Netherlands],
/record_type[]/alert / signature_id = [88006],
_attachment_body = [{
"timestamp": "2015-03-23T07:42:01.303046",
"event_type": "alert",
"src_ip": "1.1.1.1",
"src_port": 18192,
"dest_ip": "46.231.41.166",
"dest_port": 62004,
"proto": "TCP",
"alert": {
"action": "allowed",
"gid": "1",
"signature_id": "88006",
"rev": "1",
"signature" : "GeoIP from NL, Netherlands ",
"category" : ""
"severity" : "3"
}
}],
_attachment_mimetype=[json/java + memory],
basename = [simple_eve.json]
}]
UPDATE 2017-06-22
you MUST populate the data in the structure in order for this to work, by using addValues or setValues
{
addValues {
micDefaultHeader : [
{
eventTimestampString : "2017-06-22 18:18:36"
}
]
}
}
after debugging the sources of morphline toAvro, it appears that the record is the first object to be evaluated, no matter what you put in your mappings structure.
the solution is quite simple, but unfortunately took a little extra time, eclipse, running the flume agent in debug mode, cloning the source code and lots of coffee.
here it goes.
my schema:
{
"type" : "record",
"name" : "co_lowbalance_event",
"namespace" : "co.tigo.billing.cboss.lowBalance",
"fields" : [ {
"name" : "dummyValue",
"type" : "string",
"default" : "dummy"
}, {
"name" : "micDefaultHeader",
"type" : {
"type" : "record",
"name" : "mic_default_header_v_1_0",
"namespace" : "com.millicom.schemas.root.struct",
"doc" : "standard millicom header definition",
"fields" : [ {
"name" : "eventTimestampString",
"type" : "string",
"default" : "12345678910"
} ]
}
} ]
}
morphlines file:
morphlines : [
{
id : convertJsonToAvro
importCommands : ["org.kitesdk.**"]
commands : [
{
readJson {
outputClass : java.util.Map
}
}
{
addValues {
micDefaultHeader : [{}]
}
}
{
logDebug { format : "my record: {}", args : ["#{}"] }
}
{
toAvro {
schemaFile : /home/asarubbi/Development/test/co_lowbalance_event.avsc
mappings : {
"micDefaultHeader" : micDefaultHeader
"micDefaultHeader/eventTimestampString" : eventTimestampString
}
}
}
{
writeAvroToByteArray {
format : containerlessJSON
codec : null
}
}
]
}
]
the magic lies here:
{
addValues {
micDefaultHeader : [{}]
}
}
and in the mappings:
mappings : {
"micDefaultHeader" : micDefaultHeader
"micDefaultHeader/eventTimestampString" : eventTimestampString
}
explanation:
inside the code the first field name that is evaluated is micDefaultHeader of type RECORD. as there's no way to specify a default value for a RECORD (logically correct), the toAvro code evaluates this, does not get any value configured in mappings and therefore it fails at it detects (wrongly) that the record is empty when it shouldn't.
however, taking a look at the code, you may see that it requires a Map object, containing no values to please the parser and continue to the next element.
so we add a map object using the addValues and fill it with an empty map [{}]. notice that this must match the name of the record that is causing you an empty value. in my case "micDefaultHeader"
feel free to comment if you have a better solution, as this looks like a "dirty fix"