[Python] Need a better workaround for BadRequestError: cursor position is outside the range of...

This is how I deal with:

BadRequestError: cursor position is outside the range of the original query


Background: My application handles items. Each item has an owner and a datestring. It runs in two parts, first checking existing items (from previous runs), comparing them to the once stored in NDB (sets datestring). Afterwards it handles all the once without updated datesting; with the function full_run.

#runs as a deferred task
def full_run(ownerKey, run_date):
query = Item.query(ndb.AND(
Item.owner == ownerKey,
ndb.OR(Item.push_date < run_date, Item.push_date > run_date)
)).order(Item.push_date, Item._key)
more = True
curs = None
while more:
ndb_keys, curs, more = query.fetch_page(page_size=1000, start_cursor=curs, keys_only=True)
deferred.defer(handle_key_page, ndb_keys, run_date)

def handle_key_page(key_page, run_date):
for item in ndb.get_multi(key_page):
deferred.defer(do_stuff, item)
item.push_date = run_date
item.put()


Problem: When a query-item changes, the page-cursor becomes invalid. This workaround utilize that tasks rerun when errors are raised. The new run will fetch a new set of 1000 (maxage_size) entries as the updated items get excluded from the future queries.

Two issues has become painfully clear:

¤ Tasks has exponential backoff so the time it takes to finish increase exponentially. (hard to debug)

¤ I handle errors (in do_stuff) by resetting the datesting. I first thought of this as an minor issue as I thought it ok if a few items in 1000 was handled multiple times.

The two combined is making this solution less scalable than I wish it to be but I haven't been able to find a better solution.

Continue reading...