I have a problem with a script that i wrote for elasticsearch. On my server I have multiple log files that need to be deleted on a daily basis. To automate this process I wrote a Perl script that deletes my keep alive log files.
Basically an curl XDELETE
But now I want to add a query to delete another log file.
IS IT POSSIBLE TO ADD ANOTHER JSON OBJECT, WITH OUT CREATING ANOTHER DELETE VARIABLE?
So adding something to my JSON that integrates a separate queries that also deletes that log?
{
"query": {
"bool": {
"must": [
{
"range": {
"#timestamp": {
"to": "2014-08-24T00:00:00.000+01:00"
}
}
},
{
"query_string": {
"fields": [
"log_message"
],
"query": "keepAlive"
}
},
]
}
}
}
(Something Like &&? adding a second bool query)
Because everything I add will just over specify the query that i have leading to results I do not want.
Thank you
Not quite sure I've correctly understood what your looking for, but it sounds like you want to combine the results of the given query with those of some other separate query. In that case, you can nest boolean queries as should clauses, something like:
{
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"range": {
"#timestamp": {
"to": "2014-08-24T00:00:00.000+01:00"
}
}
},
{
"query_string": {
"fields": [
"log_message"
],
"query": "keepAlive"
}
},
]
}
},
{
**Another query here**
},
]
}
}
}
Related
I have an Issue that I need to wrote a elasticsearch query that give me what I look for,
first of all here is one item of my JSON object in db that query looking into:
{
"data": {
"circuit": {
"version": "2.12.2",
"createdOn": "2020-02-04T10:38:11.282",
"expirationDate": "2020-02-06T05:50:00.000",
"expiredSoonNotification": false
}
},
"createdDate": "2020-02-04T10:38:11.282"
}
What I need is to get all Items that accept this condition:
now < "data.circuit.expirationDate" < ("data.circuit.expirationDate" - "createdDate")/10 + now
meaning : I need to get all items that there expirationDate is less 10% from now
I hope that I explained my issue cause I don't know how to use fields inside lt og gt
something I did until now like that, but not working :
{
"query": {
"bool": {
"must_not": [
{
"bool": {
"must": [
{
"range": {
"data.circuit.expirationDate": {
"gt": "now",
"lt": ("data.circuit.expirationDate" - "createdDate")/10 + now
}
}
}
]
}
}
]
}
},
"sort": [
{
"createdDate": {
"order": "desc"
}
}
]
}
Thank You
You cannot do math referencing other fields in a range-query. You would need to encode your logic in a script-query using the Elasticsearch "painless" scripting-language. Script-queries are significantly slower than other queries, as the script needs to get executed for every single document. You can limit the number of documents for which the script gets executed by breaking up the logic into 2 parts:
"data.circuit.expirationDate" > now
"data.circuit.expirationDate" <
(("data.circuit.expirationDate" - "createdDate")/10 + now)
Your query structure would need to look like this (Pseudo-code):
"query": {
"bool": {
"must": { "script": "data.circuit.expirationDate" < ("data.circuit.expirationDate" - "createdDate")/10 + now) }
"filter": { "range": "data.circuit.expirationDate" > now }
}
}
You also should consider whether you really need precision down to millisecond-level. Performance-wise it would be much better to round now to a more granular unit (e.g. now/s for second-level granularity).
Pre-calculating ("data.circuit.expirationDate" - "createdDate")/10 and storing the calculated result directly in your document would furthermore increase query-performance significantly.
I get log files from my firewall which i want to filter for several strings.
However the string contains always some other information. So i want to filter the whole string for some specific words which are always in the string: "User" "authentication" "failed.
I tried this but i do not get any data from it:
"query": {
"bool": {
"must": [
{
"range": {
"#timestamp": {
"gt": "now-15m"
}
}
},
{
"query_string": {
"query": "User AND authentication AND failed"
}
}
]
}
}
}
However i cannot find the syntax for specific filtering words in strings. Hopefully some of you can help me.
This is the message log ( i want to filter "event.original"): Screenshot
Currently developing a tag search application using elasticsearch, I have given each document within the index an array of tags, here's an example of how a document looks:
_source: {
title: "Keep in touch scheme",
intro: "<p>hello this is a test</p> ",
full: " <p>again this is a test mate</p>",
media: "",
link: "/training/keep-in-touch",
tags: [
"employee",
"training"
]
}
I would like to be able to make a search and only return documents with all of the specified tags.
Using the above example, if I searched for a document with tags ["employee", "training"] then the above result would be returned.
In contrast, if I searched with tags ["employee", "other"], then nothing would be returned; all tags within the search query must match.
Currently I am doing:
query: {
bool: {
must: [
{ match: { tags: ["employee","training"] }}
]
}
}
but I am just getting returned exceptions like
IllegalStateException[Can't get text on a START_ARRAY at 1:128];
I have also tried concatenating the arrays and using comma-delimited strings, however this seems to match anything given the first tag matches.
Any suggestions on how to approach this? Cheers
Option 1: Next example should work (v2.3.2):
curl -XPOST 'localhost:9200/yourIndex/yourType/_search?pretty' -d '{
"query": {
"bool": {
"must": [
{ "term": { "tags": "employee" } } ,
{ "term": { "tags": "training" } }
]
}
}
}'
Option 2: Also you can try:
curl -XPOST 'localhost:9200/yourIndex/yourType/_search?pretty' -d '{
"query": {
"filtered": {
"query": {"match_all": {}},
"filter": {
"terms": {
"tags": ["employee", "training"]
}
}
}
}
}'
But without "minimum_should_match": 1 it works little bin not accurate.
I also found "execution": "and" but it works not accurate too.
Option 3: Also you cat try query_string it works perfectly, but looks little bit complicated:
curl -XPOST 'localhost:9200/yourIndex/yourType/_search?pretty' -d '{
"query" : {
"query_string": {
"query": "(tags:employee AND tags:training)"
}
}
}'
Maybe it will be helpful for you...
To ensure that the set contains only the specified values, maintain a secondary field to keep track of the tags count. Then you can query like below to get the desired results
"query":{
"bool":{
"must":[
{"term": {"tags": "employee"}},
{"term": {"tags": "training"}},
{"term": {"tag_count": 2}}
]
}
}
How can i convert the following sql query into elastic search query?
SELECT sum(`price_per_unit`*`quantity`) as orders
FROM `order_demormalize`
WHERE date(`order_date`)='2014-04-15'
You need to use scripts to compute the product of values. For newer versions of Elasticsearch, enable dynamic scripting by adding the line script.disable_dynamic: false in elasticsearch.yml file. Note that this may leave a security hole in your Elasticsearch cluster. So enable scripting judiciously. Try the query below:
POST <indexname>/<typename>/_search?search_type=count
{
"query": {
"filtered": {
"filter": {
"term": {
"order_date": "2014-04-15"
}
}
}
},
"aggs": {
"orders": {
"sum": {
"script": "doc['price_per_unit'].value * doc['quantity'].value"
}
}
}
}
I am trying to figure out an approach to delete all entries for a specific property in an elasticsearch index and remove all type mappings for that property.
I have been looking at the following two doc pages: put mapping and delete mapping
From second link:
"Allow to delete a mapping (type) along with its data. The REST
endpoint is /{index}/{type} with DELETE method."
What I think I need is a /{index}/{type}/{property}?
Do I need to recreate the whole index to accomplish this, i.e. moving and manipulating data between types?
For Example, calling GET on the mapping:
curl -XGET 'http://.../some_index/some_type/_mapping'
result:
{
"some_type": {
"properties": {
"propVal1": {
"type": "double",
"index": "analyzed"
},
"propVal2": {
"type": "string",
"analyzer": "keyword"
},
"propVal3": {
"type": "string",
"analyzer": "keyword"
}
}
}
}
after this delete operation on propVal3 would return:
curl -XGET 'http://.../some_index/some_type/_mapping'
result:
{
"some_type": {
"properties": {
"propVal1": {
"type": "double",
"index": "analyzed"
},
"propVal2": {
"type": "string",
"analyzer": "keyword"
}
}
}
}
and all data for propVal3 would be removed through the index.
You can not do that. Just forget that this value exists... ;-)
If you really need to remove it, you will have to reindex your documents.
You can use the new _reindex api for this, you could even PUT a new _mapping to the dest index before running the reindex so you can change the properties of the fields in your index.
To do a reindex and removing a property, you can do this:
POST /_reindex
{
"source": {
"index": "twitter"
},
"dest": {
"index": "new_twitter",
},
"script": {
"inline": "ctx._source.remove('whatever')"
}
}
if you would use this in combination with the _aliases API you can modify indexes without having any 'downtime'
It's not currently possible to remove a property from a mapping. In order to remove all values of a property from all records, you need to reindex all records with this property removed.
You can choose whats documents fields you will reindex to a new index. For example:
POST _reindex
{
"source": {
"index": "my-source-index",
"_source": ["host.hostname", "host.ip", "another_field"]
},
"dest": {
"index": "my-dest-index"
}
}
Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-filter-source
Just want to add another approach for the case of "removing property from index".
POST pulse/_update_by_query
{
"query": {
"match_all": {}
},
"script": {
"source": "ctx._source.remove(\"file_id\")",
"lang": "painless"
}
}