Is there any solution to do the "NOT IN" functionality in Elasticsearch? - mysql

Question

Here is a simple but difficult question.
I want to do an aggregation for a query results that should be use "NOT IN" functionality like any RDBMS' SQL.
For example, I want to do a job something like below.
curl -XGET http://localhost:9200/my_index/my_type/_search?pretty -d '{
"query": {
"filtered": {
"filter": {
!!! Documents whose 'user_id' field value is 'NOT IN' distinct user_ids where the 'action' field value is 'signup' !!!
}
}
},
"aggregations": {
"distinct_users":{
"cardinality": {
"field": "user_id",
"precision_threshold": 1000000
}
}
}
}'
Edit
Here is an example data.
curl -s -XPOST 'localhost:9200/my_index/my_type/1' -d'{ "user_id": 1234, "action": "signup" }'
curl -s -XPOST 'localhost:9200/my_index/my_type/2' -d'{ "user_id": 1234, "action": "visit" }'
curl -s -XPOST 'localhost:9200/my_index/my_type/3' -d'{ "user_id": 1234, "action": "visit" }'
curl -s -XPOST 'localhost:9200/my_index/my_type/4' -d'{ "user_id": 5678, "action": "visit" }'
curl -s -XPOST 'localhost:9200/my_index/my_type/5' -d'{ "user_id": 5678, "action": "visit" }'
curl -s -XPOST 'localhost:9200/my_index/my_type/6' -d'{ "user_id": 9012, "action": "signup" }'
curl -s -XPOST 'localhost:9200/my_index/my_type/7' -d'{ "user_id": 9012, "action": "visit" }'
curl -s -XPOST 'localhost:9200/my_index/my_type/8' -d'{ "user_id": 9012, "action": "visit" }'
curl -s -XPOST 'localhost:9200/my_index/my_type/9' -d'{ "user_id": 3456, "action": "visit" }'
curl -s -XPOST 'localhost:9200/my_index/my_type/10' -d'{ "user_id": 3456, "action": "visit" }'
What I really want to get is the "Documents whose user_id DOES NOT signed up based on these log data".
So, documents [4, 5, 9, 10] are the final results what I want to get.
Is it possible to get the results what I want in Elasticsearch?
Thanks in advance.

Answer 1

No, elasticsearch does not do joins and what you are asking for is a variation of a join.
As pointed out above, it does have parent child relations and nested objects though which might help you somewhat, depending on your domain.
Elasticsearch also does not have a distinct feature which you need as well. But you can sort of fake it with a terms aggregation.
However, that does not help you here since you really need a join for this. So, the only solution here is to do the join outside of elasticsearch. Depending on your data size, that might be expensive. Also see application side joins.

Answer 2

if you use the not filter Elasticsearch will check each document in turn - the following will return all documents that have an action where action is anything but signup.
curl -XGET http://localhost:9200/myindex/my_type/_search?pretty -d '{
"query": {
"filtered": {
"filter": {
"not" : {
"term" : { "action" : "signup" }
}
}
}
}
}'
To meet the requirement where a userid is returned if it doesn't have any instances of action=signup then you'll need to set up a parent child relationship.
in this case, Userid would be unique across all user type documents. Each user document would have one or more action type children.
The following query checks the action child and returns the user document.
curl -XGET 'http://localhost:9200/myindex/my_type/_search?pretty' -d '{
"query": {
"filtered": {
"filter": {
"not" : {
"has_child": { "type": "my_action", "term" : { "action" : "signup" }}
}
}
}
}
}'