[Python] Queries to Elasticsearch using Python API return different results than Kibana

I've built a set of data pipes for moving data out of Elastic into RDS, using the Python Elasticsearch library to pass a SQL query to Elasticsearch. I'm trying to add a new one that will query all the APM indices.

When I run the SQL query in the ES devtools, I get the same results as in Kibana (~140000 documents). When I run the same query (copy-pasted! checked for weird character mutations!) in Python, I get ~3500 documents. Upon investigation, it seems that the Python script is only retrieving documents from indices that errors are routed to.

In Kibana, we group all our APM indices together, like traces-apm*,apm-*,logs-apm*,apm-*,metrics-apm*,apm-*. In Kibana Discovery/Elasticsearch devtools, queries against this grouping yield 140k docs. In Python against the same group I get 3500 docs.

Any idea what the difference could be? Versions of Elastic and versions of the python elasticsearch library maybe?? I'm at my wits end y'all, this doesn't make any sense.

Continue reading...