Elasticsearch searching issues on string type - json

I'm having an issue with a search on an index. Here's my index creation:
curl -XPUT localhost:9200/my_supder_index -d '{
"mappings":
{
"doctype_I_index":
{
"properties":
{
"field_I_index":
{
"type":"string",
"term_vector":"yes"
}
}
}
}
}'
Here is a sample piece of content in that index:
{
_index:"my_super_index",
_type:"doctype_I_index",
_id:"676078",
_version:1,
found:true,
_source:{
created:"2015-05-02T00:24:03",
field_I_index:[
"21E0",
"19E0",
"5E0",
"6E0",
"4E0"
],
id:676078
}
}
Now when I do a search like this:
curl -XGET 'http://127.0.0.1:9200/my_super_index/_search' -d '{
"sort":[
{
"created":{
"order":"desc"
}
}
],
"query":{
"bool":{
"must":[
{
"terms":{
"field_I_index":[
"21E0"
],
"minimum_should_match":1
}
}
]
}
}
}'
I get zero results. It's not doing a match on the text. Can anyone point me in the right direction please?

On checking how analysis happens for this value , following are results -
curl -XPOST 'localhost:9200/news/_analyze?pretty' -d '21E0'
{
"tokens" : [ {
"token" : "21e0",
"start_offset" : 0,
"end_offset" : 4,
"type" : "<ALPHANUM>",
"position" : 1
} ]
}
Here you can see that the text is lower cased.
Also as term query does not apply analyzer for the search text , it looks for the exact match of 21E0 , but 21e0 is what is indexed.
Hence in this case , if you use match query , instead of term query , it should work.
But i would recommend to use not_analyzed to the field and then use term query on top of that. It might be a better approach.

Related

jq transform JSON resultset by parameter names (no strict order of paramaters)

I already tried a lot, without success.
I'm facing a problem with tranforming the following JSON structure:
JSON:
{
"?xml":{
"#version":"1.0",
"#encoding":"UTF-8"
},
"methodResponse":{
"params":{
"param":{
"value":{
"struct":{
"member":[
{
"name":"severity",
"value":{
"string":"Important"
}
},
{
"name":"product",
"value":{
"string":"this is a product name"
}
},
{
"name":"notes",
"value":{
"string":"Here are some notes"
}
},
{
"name":"references",
"value":{
"string":"This is a reference"
}
},
{
"name":"release",
"value":{
"i4":"1"
}
},
{
"name":"last_modified_date",
"value":{
"string":"2020-03-10 15:21:50.147896"
}
}
]
}
}
}
}
}
}
I'd like to filter it by key names like in this transform statement (which already works with one key name):
Transform statement:
map(.methodResponse.params.param.value.struct.member[] | select(.name == "product" ).value.string as $product| {$product})
But I'm not sure how the syntax must look like when I want to returning several key names in one statement.
I would expect a result like this one:
[
{
"product": "This is a product name",
"release": "1",
"last_modified_date": "2020-03-09 16:39:39.0"
}
]
I am not quite as familiar with this topic and trying it alone takes forever...
Can anyone help please? Any help is much appreciated!
Thanks!
First, if you want to use jq, you need well formated JSON data. The human JSON tool hjson might help on your input data.
hjson -j file
Extracting the wanted data could be done in the following way:
hjson -j file | \
jq '.methodResponse.params.param.value.struct.member |
[map({(.name):(.value.string//.value.i4)})|add]'
It simply creates ne object base on the name and value parameters.
Note that if .value.string doesn't exist, it uses .value.i4 with the alternate operator //.

Elasticsearch Query DSL

I get log files from my firewall which i want to filter for several strings.
However the string contains always some other information. So i want to filter the whole string for some specific words which are always in the string: "User" "authentication" "failed.
I tried this but i do not get any data from it:
"query": {
"bool": {
"must": [
{
"range": {
"#timestamp": {
"gt": "now-15m"
}
}
},
{
"query_string": {
"query": "User AND authentication AND failed"
}
}
]
}
}
}
However i cannot find the syntax for specific filtering words in strings. Hopefully some of you can help me.
This is the message log ( i want to filter "event.original"): Screenshot

Elasticsearch match all tags within given array

Currently developing a tag search application using elasticsearch, I have given each document within the index an array of tags, here's an example of how a document looks:
_source: {
title: "Keep in touch scheme",
intro: "<p>hello this is a test</p> ",
full: " <p>again this is a test mate</p>",
media: "",
link: "/training/keep-in-touch",
tags: [
"employee",
"training"
]
}
I would like to be able to make a search and only return documents with all of the specified tags.
Using the above example, if I searched for a document with tags ["employee", "training"] then the above result would be returned.
In contrast, if I searched with tags ["employee", "other"], then nothing would be returned; all tags within the search query must match.
Currently I am doing:
query: {
bool: {
must: [
{ match: { tags: ["employee","training"] }}
]
}
}
but I am just getting returned exceptions like
IllegalStateException[Can't get text on a START_ARRAY at 1:128];
I have also tried concatenating the arrays and using comma-delimited strings, however this seems to match anything given the first tag matches.
Any suggestions on how to approach this? Cheers
Option 1: Next example should work (v2.3.2):
curl -XPOST 'localhost:9200/yourIndex/yourType/_search?pretty' -d '{
"query": {
"bool": {
"must": [
{ "term": { "tags": "employee" } } ,
{ "term": { "tags": "training" } }
]
}
}
}'
Option 2: Also you can try:
curl -XPOST 'localhost:9200/yourIndex/yourType/_search?pretty' -d '{
"query": {
"filtered": {
"query": {"match_all": {}},
"filter": {
"terms": {
"tags": ["employee", "training"]
}
}
}
}
}'
But without "minimum_should_match": 1 it works little bin not accurate.
I also found "execution": "and" but it works not accurate too.
Option 3: Also you cat try query_string it works perfectly, but looks little bit complicated:
curl -XPOST 'localhost:9200/yourIndex/yourType/_search?pretty' -d '{
"query" : {
"query_string": {
"query": "(tags:employee AND tags:training)"
}
}
}'
Maybe it will be helpful for you...
To ensure that the set contains only the specified values, maintain a secondary field to keep track of the tags count. Then you can query like below to get the desired results
"query":{
"bool":{
"must":[
{"term": {"tags": "employee"}},
{"term": {"tags": "training"}},
{"term": {"tag_count": 2}}
]
}
}

How to query for different fields in elasticserch?

I want to query elasticsearch for one or two key:value pair such as:
http://localhost:9200/indexname/_search?pretty=1&q=Date:%222000-12-30T10:11:25%22&q=id:%22unique_id%22
This query takes only the unique_id into consideration. If I change the Date to any invalid value, still it gives me all values based on the Unique_id.
Any idea how to make an AND condition with both the queries? It should consider both queries and provide result accordingly? Please advice. Thanks.
According to documentation, it should work
See http://www.elasticsearch.org/guide/reference/query-dsl/query-string-query.html
That said, you can create your query as below:
http://localhost:9200/indexname/_search?pretty=1&q=%2BDate%3A2000-12-30T10:11:25+%2Bid%3Aunique_id
Note : %2B is decoded as '+' whereas '+' is decoded as ' '
Try this.
GET /index/type/_search
{
"query": {
"match": {"Date":"2015-09-17 03:45:00-04"}
},
"filter" : {
"and" : [
{
"match": {"unique_id" : "6324"}
}
]
}
}
Reference link - https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-and-query.html
You can choose this query also ,(easy way)
GET myindex/mytype/_search
{
"query": {
"bool" : {
"must" :[
{
"term" : { "unique_id" : "6324"}
},
{
"term" : { "Date":"2015-09-17 03:45:00-04"}
}
],
}
}
}

Mapping format on elasticsearch

I'm to upload a json document to my server via elasticsearch but i wanted to map it before i upload it but i keep getting a search phase execution exception error.
The json data looks like this
{"geometry":{"type":"Point","coordinates":[-73.20266100000001,45.573647]},"properties":{"persistent_id":"XVCPFsbsqB7h4PrxEtCU3w==","timestamp":1408216040000,"tower_id":"10.48.66.178"}}
So far i've tried this as my mapping. Im not sure what i am doing wrong...
curl –XPUT 'http://localhost:9200/carrier/_search?q=coordinates?pretty=true' -d'
{ “geometry”: {
“type” : {“type” : “string”},
“coordinates” : {“type” : “geo_point”}
},
“properties” : {
“persistent_id” : {“type” : “string”},
“timestamp”: { “type” : “long”},
“tower_id” : {“type” : “string”}
}'
There are a few problems here. First of all you need to use put mapping request instead of search request. The body of the request has to start with the name of the type followed by the list of properties (fields) that you add. The second problem is that you probably copied the example from some documentation where all ascii quotes (") were replaced with replaced with their fancy unicode versions (“ and ”) and dash in front of the XPUT parameter looks like n-dash – instead of normal dash -. You need to replace all fancy quotes and dashes with their ascii versions. So, all together the working statement should look like this (assuming doc as your document type):
curl -XPUT 'http://localhost:9200/carrier/doc/_mapping' -d '{
"doc": {
"properties": {
"geometry": {
"properties": {
"type": {
"type": "string"
},
"coordinates": {
"type": "geo_point"
}
}
},
"properties": {
"properties": {
"persistent_id": {
"type": "string"
},
"timestamp": {
"type": "long"
},
"tower_id": {
"type": "string"
}
}
}
}
}
}'
then you can add document like this:
curl -XPUT 'http://localhost:9200/carrier/doc/1' -d '{"geometry":{"type":"Point","coordinates":[-73.20266100000001,45.573647]},"properties":{"persistent_id":"XVCPFsbsqB7h4PrxEtCU3w==","timestamp":1408216040000,"tower_id":"10.48.66.178"}}'
Please note that in order to add the mapping you might need to delete and recreate the index if you already tried to add documents to this index and the mapping was already created.
This is because you're using the _search endpoint in order to install your mapping.
You have to use the _mapping endpoint instead, like this:
curl –XPUT 'http://localhost:9200/carrier/_mapping/geometry' -d '{
...your mapping...
}'