The first request I make should populate the data base, so I wrote something like this: # spider.py def start_requests(self): yield scrapy.Request(url=self.url, method='POST', callback=self.parse) # middlewares.py DownloaderMiddleware process_request . . . except ValueError: new_request = scrapy.Request( url=request.url, callback=request.callback, method=request.method, headers=request.headers, body=self.body # was initialized in spider_opened method ) return new_request The problem is that the request never makes it to the parse method but gets duplicated. The request is being made, the request parameters are correct, but it still somewhy doesn't get it to parse.: # spider.py def parse(self, response): logging.info('started parsing') l = ItemLoader(FeedItem()) json_response = json.loads(response.text) l.add_value('feed', json_response) yield l.load_items() yield scrapy.Request(url=self.url, method='POST', callback=self.parse) This is the traceback I'm receiving: 2024-10-05 12:31:09 [scrapy.utils.log] INFO: Scrapy 2.11.2 started (bot: feed) 2024-10-05 12:31:09 [scrapy.utils.log] INFO: Versions: lxml 5.3.0.0, libxml2 2.11.7, cssselect 1.2.0, parsel 1.9.1, w3lib 2.2.1, Twisted 24.7.0, Python 3.12.6 (tags/v3.12.6:a4a2d2b, Sep 6 2024, 20:11:23) [MSC v.1940 64 bit (AMD64)], pyOpenSSL 24.2.1 (OpenSSL 3.3.2 3 Sep 2024), cryptography 43.0.1, Platform Windows-11-10.0.22631-SP0 2024-10-05 12:31:09 [scrapy.addons] INFO: Enabled addons: [] 2024-10-05 12:31:09 [asyncio] DEBUG: Using selector: SelectSelector 2024-10-05 12:31:09 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor 2024-10-05 12:31:09 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.windows_events._WindowsSelectorEventLoop 2024-10-05 12:31:09 [scrapy.extensions.telnet] INFO: Telnet Password: 1eee8686399dba16 2024-10-05 12:31:09 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.logstats.LogStats'] 2024-10-05 12:31:09 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'feed', 'CONCURRENT_REQUESTS': 1, 'DOWNLOAD_DELAY': 80, 'FEED_EXPORT_ENCODING': 'utf-8', 'NEWSPIDER_MODULE': 'feed.spiders', 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 'SPIDER_MODULES': ['feed.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'} 2024-10-05 12:31:10 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'feed.middlewares.FeedDownloaderMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2024-10-05 12:31:10 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2024-10-05 12:31:10 [scrapy.middleware] INFO: Enabled item pipelines: ['feed.pipelines.FeedPipeline'] 2024-10-05 12:31:10 [scrapy.core.engine] INFO: Spider opened 2024-10-05 12:31:10 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2024-10-05 12:31:10 [feed] INFO: Spider opened: feed 2024-10-05 12:31:10 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024 2024-10-05 12:31:10 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <POST url> (referer: url) 2024-10-05 12:31:10 [scrapy.core.engine] INFO: Closing spider (finished) 2024-10-05 12:31:10 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'dupefilter/filtered': 1, 'elapsed_time_seconds': 0.023085, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2024, 10, 5, 10, 31, 10, 174051, tzinfo=datetime.timezone.utc), 'log_count/DEBUG': 4, 'log_count/INFO': 11, 'scheduler/dequeued': 2, 'scheduler/dequeued/memory': 2, 'scheduler/enqueued': 2, 'scheduler/enqueued/memory': 2, 'start_time': datetime.datetime(2024, 10, 5, 10, 31, 10, 150966, tzinfo=datetime.timezone.utc)} 2024-10-05 12:31:10 [scrapy.core.engine] INFO: Spider closed (finished) EDIT: def start_requests(self): yield scrapy.Request(url=self.url, method='POST', callback=self.parse, flags=['this is the start request method']) traceback Continue reading...
