ElasticSearch: Aggregation with order by top_hits differs in doc count - json

Question

My problem is that having a max score aggregation that sorts by _score, in combination with a terms aggregation, is showing a different doc_count.
Here is the aggregation block that I want, but giving a lower doc_count for certain results. Again, it always gives a lower value; never higher.
"aggs": {
"group_by_field": {
"terms": {
"field": "field4",
"order": {
"max_score": "desc"
}
},
"aggs": {
"max_score": {
"max": {
"script": "_score"
}
}
}
}
}
Here is the aggregation block that I do not want, but is showing the correct doc_count for all results.
"aggs": {
"group_by_target": {
"terms": {
"field": "target"
}
}
}
For those interested, here is my full body query:
{
"index": "myIndex",
"type": "myType",
"body": {
"query": {
"bool": {
"filter": {
0: {
"term": {
"term1": false,
}
}
},
"must": {
"bool": {
"should": {
0: {
"match_phrase": {
"phrase1": "example",
}
}
},
"minimum_should_match": 1,
}
}
}
},
"size": 0,
"_source": {
0: "field1",
1: "field2",
2: "field3",
3: "field4",
},
"aggs": {
"group_by_field": {
"terms": {
"field": "field4",
"size": 25,
"order": {
"max_score": "desc",
}
},
"aggs": {
"max_score": {
"max": {
"script": "_score",
}
},
"my_top_agg": {
"top_hits": {
"size": 3,
"_source": {
0: "field1",
1: "field2",
2: "field3",
3: "field4",
}
}
}
}
},
"my_total": {
"cardinality": {
"field": "field4",
}
}
}
}
}
I have test both queries and each time it is showing a the same incorrect doc_count when having the order aggregation. What is causing the problem?