[Python] A way to defer yielding a Request in scrapy?

My scrapy logic is as follows:

1. get all rows from child_page_table where parent_page_id is null
2. for each row, if parent_page_id is (still) null, yield a Request with callback scrape_page
3. [scrape_page] if this a "parent page," add data to parent_page_table and get a list of all "child pages" linked from the parent page. Insert those into child_page_table with parent_page_id pointing to what we just inserted to parent_page_table.
4. if this is not a parent page, yield Request of the parent page linked within the child page with callback scrape_page (which upon recursing will run step #3 rather than step #4)

The logic here is that I've got tens of thousands of child pages and parent pages point to many of those child pages. The naive way of filling in the parent_page_id is just for every child page, request it and fetch the parent page and request that, etc.

But a more efficient way is get the first child page, fetch the parent page, and the parent page will have dozens or hundreds of child pages linked. For each of those, making no more requests, update the data in the child table so that we never have to make a Request for those children.

However, it appears that what scrapy's start does is yield all 15,000 rows-as-Requests immediately rather than what I'd assumed, which was it would yield 16-ish at a time (the number of concurrent requests I'm allowing).

So essentially step #2 always happens as it happens immediately after step #1 and no requests have been called to update any child rows' parent_page_id.

Is there a way to create a Request that will, right before making the request, re-check if the request is still necessary? Essentially delay the "is it necessary?" logic until immediately before that request is made?

Possibly some way to thunk the request until whatever manages the requests is actually going to fire it off to HTTP land?

children = list(enumerate(self.cur.execute("select tag from children where parent_page_id is null;").fetchall()))
rel_len = len(relationships)
for (idx, row) in relationships:
logging.debug("[Children] Fetching " + str(idx) + "/" + str(rel_len))
child_item = Page(tag=row[0], type=TagType.CHILD)
if self._needs_call(child_item) // checks if parent_page_id is null
yield Request(
base_url + tag_item.tag,
callback=self.parse_page,
cb_kwargs=dict(tag=row[0], tag_type=TagType.CHILD),
errback=self.handle_error,
meta=MyScraper.meta
)

Continue reading...