With PyMongo 3.7.2 I'm trying to read a collection in chunks by using batch_size on the MongoDB cursor, as described here. The basic idea is to use the find() method on the collection object, with batch_size as parameter. But whatever I try, the cursor always returns all documents in my collection. A basic snippet of my code looks like this (the collection has over 10K documents): import pymongo as pm client = pm.MongoClient() coll = client.get_database('db').get_collection('coll') cur = coll.find({}, batch_size=500) However, the cursor always returns the full collection size immediately. I'm using it as described in the docs. Does anyone have an idea how I would properly iterate over the collection in batches? There are ways to loop over the output of the find() method, but that would still get the full collection first, and would only loop over the already pulled documents in memory. The batch_size parameter is supposed to get a batch and make a round-trip every time to the server, to save memory space. Continue reading...
