I'm trying to scrape a rather large website (with around 1 million pages) with Scrapy. The spider works fine and it is able to scrape a few thousand pages before inevitably crashing due to low memory. Things I've tried: Using the -s JOBDIR=<DIRECTORY>: This gave me an initial improvement and I was able to crawl about twice the number of URLs than with the previous approach. However, even with this option Scrapy's memory consumption slowly increases, until it is killed by the out-of-memory killer. Preventing unnecessary functions, such as preventing excessive output by raising the log limit from DEBUG to INFO. Using yield statements instead of returning arrays. Keeping the returned data to an absolute minimum. Running the spider on a beefier machine: This helps me crawl a bit more, but inevitably it crashes again at a later point (and I'm nowhere near the 1 million mark). Is there something I'm missing which can help me with complete the scraping? Continue reading...
