I have a requirement for transforming JSON and I am trying to use the same value multiple times. Is there a way to use value multiple times previously I did use in array but this time I have to go through the level. Any help is appreciated and thank you.
Note I want to filter product configurations based on the name.
How to use same field value at multiple places in Jolt
Input:
{
"sample": {
"Result": {
"value": {
"ImplProductConfigurationsOptionsEcomResponse": {
"transactionId": "12345678",
"timeStamp": 1646087177771,
"orderCorrelationId": "11039714",
"implValidateMultiProductConfigurationsResponse": {
"productConfigurations": [
{
"productConfiguration": {
"productSpecification": {
"name": "Private",
"id": "247541"
}
}
},
{
"productConfiguration": {
"productSpecification": {
"name": " sample Miscellaneous",
"id": "22222"
}
}
}
]
}
}
}
}
}
}
Current output:
{
"productConfigurations": {
"productConfiguration": {
"productSpecification": {
"name": "Private",
"id": "247541"
}
}
}
}
Expected Output:
{
"productConfigurations": {
"productConfiguration": {
"productSpecification": {
"name": "Private",
"id": "247541"
}
}
},
"Details": {
"ImplProductConfigurationsOptionsEcomResponse": {
"transactionId": "12345678",
"timeStamp": 1646087177771,
"orderCorrelationId": "11039714",
"implValidateMultiProductConfigurationsResponse": {
"productConfigurations": [
{
"productConfiguration": {
"productSpecification": {
"name": "Private",
"id": "247541"
}
}
},
{
"productConfiguration": {
"productSpecification": {
"name": " sample Miscellaneous",
"id": "22222"
}
}
}
]
}
}
}
}
My Spec:
[
{
"operation": "shift",
"spec": {
"sample": {
"Result": {
"value": {
"ImplProductConfigurationsOptionsEcomResponse": {
"implValidateMultiProductConfigurationsResponse": {
"productConfigurations": {
"*": {
"productConfiguration": {
"productSpecification": {
"name": {
"*Miscellaneous": null,
"*": {
"#4": "productConfigurations"
}
}
}
}
}
}
}
}
}
}
}
}
}
]
You can reference the object twice along with a shift transformation spec such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"*": {
"*": "Details.&1.&",
"implValidateMultiProductConfigurationsResponse": {
"*": {
"*": {
"*": {
"*": {
"name": {
"*Miscellaneous": {
"#3": "Details.&8.&7.&6[].&4"
},
"*": {
"#4": "&6",
"#3": "Details.&8.&7.&6[].&4"
}
}
}
}
}
}
}
}
}
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/
i have a probleme with elastic search when i request it with the following json, i have this error: [bool] malformed query, expected [END_OBJECT] but found [FIELD_NAME]
i've tried to look on many site but none of them give me a response
{
"query":{
"bool":{
"must":[
{
"match":{
"group_issuer_name":"bnp"
}
},
{
"match":{
"asset_country":"France"
}
}
]
},
"aggs":{
"by_ptf_name":{
"terms":{
"field":"ptf_name.keyword"
},
"aggs":{
"by_ptf_id":{
"terms":{
"field":"ptf_id.keyword"
},
"aggs":{
"sum_of_asset_exposure":{
"sum":{
"field":"asset_exposure"
}
},
"min_of_ptf_total_asset":{
"min":{
"field":"ptf_total_asset"
}
}
}
}
}
}
}
}
}
You are missing }. The query part must be closed and then the aggregation part should start.
The structure should be
{
"query": {},
"aggregation": {}
}
Modify your query as -
{
"query": {
"bool": {
"must": [
{
"match": {
"group_issuer_name": "bnp"
}
},
{
"match": {
"asset_country": "France"
}
}
]
}
}, // note this
"aggs": {
"by_ptf_name": {
"terms": {
"field": "ptf_name.keyword"
},
"aggs": {
"by_ptf_id": {
"terms": {
"field": "ptf_id.keyword"
},
"aggs": {
"sum_of_asset_exposure": {
"sum": {
"field": "asset_exposure"
}
},
"min_of_ptf_total_asset": {
"min": {
"field": "ptf_total_asset"
}
}
}
}
}
}
}
}
My aim is to send 3 separate queries in one ES request using multiple search
I am using NEST client to send query to Elastic search using function below
IElasticClient _elasticClient.LowLevel.Msearch<string>(query).Body;
Passing as a raw query using curl command works absolutely fine, but NEST MSearch only returns "event_results" and "venue_results" but not "location_results"
curl -XPOST localhost:9200/_msearch -d '
{"index" : "search_results"}
{ "size": 0, "query": { "bool": { "must": [ { "term": { "partnersites": "16" } }, { "match_phrase_prefix": { "name": "manchester" } } ] } }, "aggs": { "event_results": { "terms": { "field": "name.keyword", "size": 1 }, "aggs": { "top_tag_hits": { "top_hits": { "size": 1, "_source": [ "name", "groupedName", "groupedDisplayName", "groupedUrl", "eventCode", "venueName", "venueTown", "venueId", "venueUrl", "media", "categories.name" ] } } } } } }
{"index" : "search_results2"}
{ "size": 0, "query": { "bool": { "must": [ { "term": { "partnersites": "16" } }, { "match_phrase_prefix": { "venueName": "Manchester" } } ] } }, "aggs": { "venue_results": { "terms": { "field": "name.keyword", "size": 1 }, "aggs": { "top_tag_hits": { "top_hits": { "size": 1, "_source": [ "name", "groupedName", "groupedDisplayName", "groupedUrl", "eventCode", "venueName", "venueTown", "venueId", "venueUrl", "media", "categories.name" ] } } } } } }
{"index" : "search_results3"}
{ "size": 0, "query": { "bool": { "must": [ { "term": { "partnersites": "16" } }, { "match_phrase_prefix": { "venueTown": "manchester" } } ] } }, "aggs": { "location_results": { "terms": { "field": "name.keyword", "size": 1 }, "aggs": { "top_tag_hits": { "top_hits": { "size": 1, "_source": [ "name", "groupedName", "groupedDisplayName", "groupedUrl", "eventCode", "venueName", "venueTown", "venueId", "venueUrl", "media", "categories.name" ] } } } } } }
'
Is any of you know where the issue could be?
Whenever possible, you don't want to use the "LowLevel" stuff. Instead, use what is available to you on the IElasticClient. Here is an example of how to use IElasticClient.MultiSearch to run 3 searches using the fluent syntax (which is the preferred way to do this).
var mSearchResponse = ElasticClient.MultiSearch(msearch => msearch
.Search<RedemptionES>(
s1 => s1.Query(
q=>q.Term(
t=> t.OnField(f=> f.Id).Value("123")
)
)
)
.Search<RedemptionES>(
s2 => s2.Query(
q => q.Term(
t => t.OnField(f => f.Id).Value("456")
)
)
)
.Search<RedemptionES>(
s3 => s3.Query(
q => q.Term(
t => t.OnField(f => f.Id).Value("789")
)
)
)
);
Issue lies on indention. Try this:
public static string qu = #"{""index"" : ""search_results""}
{ ""size"": 0, ""query"": { ""bool"": { ""must"": [ { ""term"": { ""partnersites"": ""16"" } }, { ""match_phrase_prefix"": { ""name"": ""manchester"" } } ] } }, ""aggs"": { ""event_results"": { ""terms"": { ""field"": ""name.keyword"", ""size"": 1 }, ""aggs"": { ""top_tag_hits"": { ""top_hits"": { ""size"": 1, ""_source"": [ ""name"", ""groupedName"", ""groupedDisplayName"", ""groupedUrl"", ""eventCode"", ""venueName"", ""venueTown"", ""venueId"", ""venueUrl"", ""media"", ""categories.name"" ] } } } } } }
{""index"" : ""search_results2""}
{ ""size"": 0, ""query"": { ""bool"": { ""must"": [ { ""term"": { ""partnersites"": ""16"" } }, { ""match_phrase_prefix"": { ""venueName"": ""Manchester"" } } ] } }, ""aggs"": { ""venue_results"": { ""terms"": { ""field"": ""name.keyword"", ""size"": 1 }, ""aggs"": { ""top_tag_hits"": { ""top_hits"": { ""size"": 1, ""_source"": [ ""name"", ""groupedName"", ""groupedDisplayName"", ""groupedUrl"", ""eventCode"", ""venueName"", ""venueTown"", ""venueId"", ""venueUrl"", ""media"", ""categories.name"" ] } } } } } }
{""index"" : ""search_results3""}
{ ""size"": 0, ""query"": { ""bool"": { ""must"": [ { ""term"": { ""partnersites"": ""16"" } }, { ""match_phrase_prefix"": { ""venueTown"": ""manchester"" } } ] } }, ""aggs"": { ""location_results"": { ""terms"": { ""field"": ""name.keyword"", ""size"": 1 }, ""aggs"": { ""top_tag_hits"": { ""top_hits"": { ""size"": 1, ""_source"": [ ""name"", ""groupedName"", ""groupedDisplayName"", ""groupedUrl"", ""eventCode"", ""venueName"", ""venueTown"", ""venueId"", ""venueUrl"", ""media"", ""categories.name"" ] } } } } } }
";
var result = _elasticClient.LowLevel.Msearch<string>(qu).Body; // Query ES for results.
I have an elasticsearch query:
"query": {
"bool": {
"minimum_should_match": 1,
"should": {
"span_not": {
"exclude": {
"span_or": {
"clauses": [{
"span_near": {
...
},
"span_near": {
...
}
}]
}
}...
I need to have in include clause something like:
"span_or": {
clauses: [
{
"span_near": {...}
}
,
{
"span_and": {
"clases": [
{
"span_near": {...}
}
,
{
"span_near": {...}
}
,
...
]
}
}
]
}
I mean, that i have "or" query:
(span_near or span_near or span_near or ...)
And i want to get "and" query:
(span_near or (span_near and span_near) or ...)
How can i do it? There is no tag "span_and". What tag can i you instead?
Update 1
I have tried this:
"span_or": {
clauses: [
{
"span_multi": {
"match": {
"regexp": {
"message": "путин.*"
}
}
}
}
,
{
"span_multi": {
"match": {
"bool": {
"must": [
{
"term" : { "message" : "test" }
},
{
"term" : { "message" : "rrr" }
}
]
}
}
}
}
]
}
but i have a error:
spanMultiTerm [match] must be of type multi term query
Would normal bool queries solve your problem?
{
"query": {
"bool": {
"must_not": {
"bool": {
"should": [
{
"regexp":{
"message": "путин.*"
}
},
{
"bool": {
"must": [
{
"span_near": {...}
},
{
"span_near": {...}
}
]
}
}
]
}
}
}
}
}
I using the following idea found here:
In certain situations, it can be convenient to have a SpanAndQuery. You can easily simulate this using a SpanNearQuery with a distance of Integer.MAX_VALUE.
"span_or": {
clauses: [
{
"span_multi": {
"match": {
"regexp": {
"message": "путин.*"
}
}
}
}
,
{
"span_near": {
"clauses": [
{
"span_term" : { "message" : "test" }
},
{
"span_term" : { "message" : "rrr" }
}
],
"slop": 2147483647,
"in_order": false
}
}
]
}
I have a json object with system environments. I need to list out that percolator if environment variable does not contain a variable name "JAVA_HOME" and also if JAVA_HOME's value is not present in PATH variable. Is this possible in Elasticsearch?
below is my percolator
PUT /eg/.percolator/2
{
"query": {
"filtered": {
"query": {
"bool": {
"must": [
{
"match": {
"client.name": "Athena"
}
},
{
"match": {
"client.environment.variable.#name": "JAVA_HOME"
}
}
]
}
}
}
}
}
my doc
GET /eg/message/_percolate
{
"doc": {
"client": {
"name": "Athena",
"environment": {
"variable": [
{
"#name": "JAVA_HOME",
"#value": "/home/vikrant/Linux/Sandra/java"
},
{
"#name": "PATH",
"#value": "/bin:/sbin:/usr/bin:/usr/sbin:/home/vikrant/Linux/FAS611:/home/vikrant/Linux/FAS611/odbc:/home/vikrant/Linux/Sandra/java/bin:/home/vikrant/Linux/Sandra/server/bin:.:/bin:/usr/bin:/sbin:/usr/sbin:/usr/bin/X11:/usr/local/bin:/usr/local/etc:/etc:/usr/etc:.:/databases/ORACLE/bin:/databases/SYBASE/OCS-15_0/bin:/databases/DB2/bin:/usr/odbc/bin"
}
]
}
}
}
}
You need nested mapping first since your docs have nesting. This is the mapping that I first created for your docs to make things work:
{
"nested_exp": {
"properties": {
"client": {
"type": "nested",
"properties": {
"environment": {
"type": "nested",
"properties": {
"variable": {
"type": "nested",
"properties": {
"#name": {
"type": "string"
},
"#value": {
"type": "string"
}
}
}
}
},
"name": {
"type": "string"
}
}
}
}
}
}
Then I changed your provided query to make it work as you expected:
{
"query": {
"filtered": {
"query": {
"bool": {
"must": [
{
"nested": {
"path": "client",
"query": {
"match": {
"client.name": "Athena"
}
}
}
},
{
"nested": {
"path": "client.environment.variable",
"query": {
"match": {
"client.environment.variable.#name": "JAVA_HOME"
}
}
}
}
]
}
}
}
}
}
Coming to your question, it has two parts.
For the first part, you can use the NOT filter with nested match filter. Something like this:
{
"query": {
"filtered": {
"query": {
"bool": {
"must": [
{
"nested": {
"path": "client",
"query": {
"match": {
"client.name": "Athena"
}
}
}
},
{
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"not": {
"filter": {
"nested": {
"path": "client.environment.variable",
"query": {
"match": {
"client.environment.variable.#name": "JAVA_HOME"
}
}
}
}
}
}
}
}
]
}
}
}
}
}
I just tried on my local Elasticsearch server and it works like a charm.
The second one seems difficult to implement with Elasticsearch only as what you what to query is something like value of 1 variable should not be present in another variable's value.