Elasticsearch "span and" - json

I have an elasticsearch query:
"query": {
"bool": {
"minimum_should_match": 1,
"should": {
"span_not": {
"exclude": {
"span_or": {
"clauses": [{
"span_near": {
...
},
"span_near": {
...
}
}]
}
}...
I need to have in include clause something like:
"span_or": {
clauses: [
{
"span_near": {...}
}
,
{
"span_and": {
"clases": [
{
"span_near": {...}
}
,
{
"span_near": {...}
}
,
...
]
}
}
]
}
I mean, that i have "or" query:
(span_near or span_near or span_near or ...)
And i want to get "and" query:
(span_near or (span_near and span_near) or ...)
How can i do it? There is no tag "span_and". What tag can i you instead?
Update 1
I have tried this:
"span_or": {
clauses: [
{
"span_multi": {
"match": {
"regexp": {
"message": "путин.*"
}
}
}
}
,
{
"span_multi": {
"match": {
"bool": {
"must": [
{
"term" : { "message" : "test" }
},
{
"term" : { "message" : "rrr" }
}
]
}
}
}
}
]
}
but i have a error:
spanMultiTerm [match] must be of type multi term query

Would normal bool queries solve your problem?
{
"query": {
"bool": {
"must_not": {
"bool": {
"should": [
{
"regexp":{
"message": "путин.*"
}
},
{
"bool": {
"must": [
{
"span_near": {...}
},
{
"span_near": {...}
}
]
}
}
]
}
}
}
}
}

I using the following idea found here:
In certain situations, it can be convenient to have a SpanAndQuery. You can easily simulate this using a SpanNearQuery with a distance of Integer.MAX_VALUE.
"span_or": {
clauses: [
{
"span_multi": {
"match": {
"regexp": {
"message": "путин.*"
}
}
}
}
,
{
"span_near": {
"clauses": [
{
"span_term" : { "message" : "test" }
},
{
"span_term" : { "message" : "rrr" }
}
],
"slop": 2147483647,
"in_order": false
}
}
]
}

Related

Python generate json: bool malformed query, expected END_OBJECT but found FIELD_NAME

i have a probleme with elastic search when i request it with the following json, i have this error: [bool] malformed query, expected [END_OBJECT] but found [FIELD_NAME]
i've tried to look on many site but none of them give me a response
{
"query":{
"bool":{
"must":[
{
"match":{
"group_issuer_name":"bnp"
}
},
{
"match":{
"asset_country":"France"
}
}
]
},
"aggs":{
"by_ptf_name":{
"terms":{
"field":"ptf_name.keyword"
},
"aggs":{
"by_ptf_id":{
"terms":{
"field":"ptf_id.keyword"
},
"aggs":{
"sum_of_asset_exposure":{
"sum":{
"field":"asset_exposure"
}
},
"min_of_ptf_total_asset":{
"min":{
"field":"ptf_total_asset"
}
}
}
}
}
}
}
}
}
You are missing }. The query part must be closed and then the aggregation part should start.
The structure should be
{
"query": {},
"aggregation": {}
}
Modify your query as -
{
"query": {
"bool": {
"must": [
{
"match": {
"group_issuer_name": "bnp"
}
},
{
"match": {
"asset_country": "France"
}
}
]
}
}, // note this
"aggs": {
"by_ptf_name": {
"terms": {
"field": "ptf_name.keyword"
},
"aggs": {
"by_ptf_id": {
"terms": {
"field": "ptf_id.keyword"
},
"aggs": {
"sum_of_asset_exposure": {
"sum": {
"field": "asset_exposure"
}
},
"min_of_ptf_total_asset": {
"min": {
"field": "ptf_total_asset"
}
}
}
}
}
}
}
}

usage of yang model ietf-routing

I am trying to add config data according to these yang modules:
https://github.com/mbj4668/pyang/blob/master/modules/ietf/ietf-routing.yang
https://github.com/mbj4668/pyang/blob/master/modules/ietf/ietf-ipv4-unicast-routing.yang
I am getting error sysrepocfg error: libyang: Unknown element "next-hop-list" when trying to use "next-hop-list" with below data.
{
"ietf-routing:routing": {
"control-plane-protocols": {
"control-plane-protocol": [
{
"type": "static",
"name": "static-routing-protocol",
"static-routes": {
"ietf-ipv4-unicast-routing:ipv4": {
"route": [
{
"destination-prefix": "0.0.0.0/0",
"next-hop-list": {
"next-hop": [
{
"index": "1",
"next-hop-address": "192.0.2.2"
}
]
}
}
]
}
}
}
]
}
}
}
Unable to figure out the error, any help?
I am able to use "simple-next-hop" with below data, that works fine.
{
"ietf-routing:routing": {
"control-plane-protocols": {
"control-plane-protocol": [
{
"type": "static",
"name": "static-routing-protocol",
"static-routes": {
"ietf-ipv4-unicast-routing:ipv4": {
"route": [
{
"destination-prefix": "0.0.0.0/0",
"next-hop": {
"next-hop-address": "192.0.2.2"
}
}
]
}
}
}
]
}
}
}
Fixed it! 'next-hop-list' had to be inside 'next-hop'.
{
"ietf-routing:routing": {
"control-plane-protocols": {
"control-plane-protocol": [
{
"type": "static",
"name": "static-routing-protocol",
"static-routes": {
"ietf-ipv4-unicast-routing:ipv4": {
"route": [
{
"destination-prefix": "0.0.0.0/0",
"next-hop": {
"next-hop-list": {
"next-hop": [
{
"index": "1",
"next-hop-address": "192.0.2.2"
},
{
"index": "2",
"next-hop-address": "192.0.2.3"
}
]
}
}
}
]
}
}
}
]
}
}
}

ElasticSearch Nest Msearch fn does not bring all results

My aim is to send 3 separate queries in one ES request using multiple search
I am using NEST client to send query to Elastic search using function below
IElasticClient _elasticClient.LowLevel.Msearch<string>(query).Body;
Passing as a raw query using curl command works absolutely fine, but NEST MSearch only returns "event_results" and "venue_results" but not "location_results"
curl -XPOST localhost:9200/_msearch -d '
{"index" : "search_results"}
{ "size": 0, "query": { "bool": { "must": [ { "term": { "partnersites": "16" } }, { "match_phrase_prefix": { "name": "manchester" } } ] } }, "aggs": { "event_results": { "terms": { "field": "name.keyword", "size": 1 }, "aggs": { "top_tag_hits": { "top_hits": { "size": 1, "_source": [ "name", "groupedName", "groupedDisplayName", "groupedUrl", "eventCode", "venueName", "venueTown", "venueId", "venueUrl", "media", "categories.name" ] } } } } } }
{"index" : "search_results2"}
{ "size": 0, "query": { "bool": { "must": [ { "term": { "partnersites": "16" } }, { "match_phrase_prefix": { "venueName": "Manchester" } } ] } }, "aggs": { "venue_results": { "terms": { "field": "name.keyword", "size": 1 }, "aggs": { "top_tag_hits": { "top_hits": { "size": 1, "_source": [ "name", "groupedName", "groupedDisplayName", "groupedUrl", "eventCode", "venueName", "venueTown", "venueId", "venueUrl", "media", "categories.name" ] } } } } } }
{"index" : "search_results3"}
{ "size": 0, "query": { "bool": { "must": [ { "term": { "partnersites": "16" } }, { "match_phrase_prefix": { "venueTown": "manchester" } } ] } }, "aggs": { "location_results": { "terms": { "field": "name.keyword", "size": 1 }, "aggs": { "top_tag_hits": { "top_hits": { "size": 1, "_source": [ "name", "groupedName", "groupedDisplayName", "groupedUrl", "eventCode", "venueName", "venueTown", "venueId", "venueUrl", "media", "categories.name" ] } } } } } }
'
Is any of you know where the issue could be?
Whenever possible, you don't want to use the "LowLevel" stuff. Instead, use what is available to you on the IElasticClient. Here is an example of how to use IElasticClient.MultiSearch to run 3 searches using the fluent syntax (which is the preferred way to do this).
var mSearchResponse = ElasticClient.MultiSearch(msearch => msearch
.Search<RedemptionES>(
s1 => s1.Query(
q=>q.Term(
t=> t.OnField(f=> f.Id).Value("123")
)
)
)
.Search<RedemptionES>(
s2 => s2.Query(
q => q.Term(
t => t.OnField(f => f.Id).Value("456")
)
)
)
.Search<RedemptionES>(
s3 => s3.Query(
q => q.Term(
t => t.OnField(f => f.Id).Value("789")
)
)
)
);
Issue lies on indention. Try this:
public static string qu = #"{""index"" : ""search_results""}
{ ""size"": 0, ""query"": { ""bool"": { ""must"": [ { ""term"": { ""partnersites"": ""16"" } }, { ""match_phrase_prefix"": { ""name"": ""manchester"" } } ] } }, ""aggs"": { ""event_results"": { ""terms"": { ""field"": ""name.keyword"", ""size"": 1 }, ""aggs"": { ""top_tag_hits"": { ""top_hits"": { ""size"": 1, ""_source"": [ ""name"", ""groupedName"", ""groupedDisplayName"", ""groupedUrl"", ""eventCode"", ""venueName"", ""venueTown"", ""venueId"", ""venueUrl"", ""media"", ""categories.name"" ] } } } } } }
{""index"" : ""search_results2""}
{ ""size"": 0, ""query"": { ""bool"": { ""must"": [ { ""term"": { ""partnersites"": ""16"" } }, { ""match_phrase_prefix"": { ""venueName"": ""Manchester"" } } ] } }, ""aggs"": { ""venue_results"": { ""terms"": { ""field"": ""name.keyword"", ""size"": 1 }, ""aggs"": { ""top_tag_hits"": { ""top_hits"": { ""size"": 1, ""_source"": [ ""name"", ""groupedName"", ""groupedDisplayName"", ""groupedUrl"", ""eventCode"", ""venueName"", ""venueTown"", ""venueId"", ""venueUrl"", ""media"", ""categories.name"" ] } } } } } }
{""index"" : ""search_results3""}
{ ""size"": 0, ""query"": { ""bool"": { ""must"": [ { ""term"": { ""partnersites"": ""16"" } }, { ""match_phrase_prefix"": { ""venueTown"": ""manchester"" } } ] } }, ""aggs"": { ""location_results"": { ""terms"": { ""field"": ""name.keyword"", ""size"": 1 }, ""aggs"": { ""top_tag_hits"": { ""top_hits"": { ""size"": 1, ""_source"": [ ""name"", ""groupedName"", ""groupedDisplayName"", ""groupedUrl"", ""eventCode"", ""venueName"", ""venueTown"", ""venueId"", ""venueUrl"", ""media"", ""categories.name"" ] } } } } } }
";
var result = _elasticClient.LowLevel.Msearch<string>(qu).Body; // Query ES for results.

Elasticsearch trying to query term and time range

I am trying to find all documents in which the content field contains the word "syria" and have the epoch time be greater than 1465312440000. The following query runs, but does only return the documents that contain word "syria". How do I fix this?(Elasticsearch version 2.2)
{
"query": {
"filtered": {
"query": {
"match": {
"content": "syria"
},
"filter": {
"term": {
"sourceOriginator": "Twitter"
},
"bool": {
"range": {
"epochCollectionDate": {
"gte": 1465312440
}
}
}
}
}
}
}
}
Thank you Guys.
I struggled with this so if someone is looking for how to do this with aggregation as well, i used winlogbeat but it will work with other indexes just change terms and field names.
I tested this with Elastic 7.1.1
{
"size": 0,
"query": {
"bool": {
"must": [
{
"match": {
"winlog.event_id": "5156"
}
}
],
"filter": [
{
"term": {
"winlog.provider_name" : "Microsoft-Windows-Security-Auditing"
}
},
{
"range": {
"#timestamp": {
"gt": "now-10d",
"lt": "now"
}
}
}
]
}
},
"aggregations": {
"event_count": {
"value_count": {
"field": "winlog.event_id"
}
},
"group_by_host": {
"terms": {
"field": "host.name",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
},
}
}
Of course it is hard to test without the data, but the filter is wrong. It should be on the same level as the second query. To my opinion the following solution is easier:
{
"query": {
"bool": {
"must": [
{
"match": {
"content": "syria"
}
}
],
"filter": [
{
"term": {
"sourceOriginator": "Twitter"
}
},
{
"range": {
"epochCollectionDate": {
"gte": 1465312440
}
}
}
]
}
}
}
Just to complement #Jettro's solution which will only work on ES 2.0 and later, the following one will work on all versions up to ES 5.
{
"query": {
"filtered": {
"query": {
"match": {
"content": "syria"
}
},
"filter": {
"bool": {
"must": [
{
"term": {
"sourceOriginator": "Twitter"
}
},
{
"range": {
"epochCollectionDate": {
"gte": 1465312440
}
}
}
]
}
}
}
}
}
Note that if you are on ES 2.0 or later, you should really use #Jettro's solution as the filtered query has been deprecated in 2.0.

Parse error nested object for Elasticsearch

I receive the following error during a search on Elasticsearch.
QueryParsingException[[dev_app] [nested] nested object under path
[contactNames] is not of nested type];
While checking the actual documentation the below request object is valid
https://www.elastic.co/guide/en/elasticsearch/reference/1.6/query-dsl-nested-query.html
This is the request object:
[
{
"query": {
"bool": {
"must": [
{
"constant_score": {
"query": {
"match": {
"contactBookId": {
"query": 15496
}
}
}
}
},
{
"constant_score": {
"query": {
"nested": {
"path": "contactNames",
"query": {
"bool": {
"must": [
{
"match": {
"contactNames.fullName": {
"query": "fewafwa"
}
}
},
{
"match": {
"contactNames.nameIndex": {
"query": "1"
}
}
}
]
}
}
}
}
}
}
]
}
},
"size": 100
}
]
Thanks for the help
The issue was that the indexes where not populated in Elasticsearch. Therefor the error was a bit odd
Review your index mappings, contactNames needs to be nested type.
It should look like this (adopt it to your needs):
{
"mappings": {
"yourType": {
"properties": {
"contactNames": {
"type": "nested",
"properties": {
"fullName": {
"type": "string"
},
"nameIndex": {
"type": "integer"
}
}
}
}
}
}
}
Then it's not going to throw you an exception and work as expeted.