[Python] Scrapy: "RuntimeError: Engine Not Running" when I try to run my spider after...

Discussão em 'Python' iniciado por Stack, Abril 22, 2025.

  1. Stack

    Stack Membro Participativo

    Background: I just installed scrapy-playwright on my virtual environment in order to scrape a website that renders some links I need with Javascript. The installation went well, but when I ran my spider in the terminal (which previously would begin crawling without major issues), it would open and the cursor would hang with no output. When I stopped it via Ctrl+C, it would throw the following exception: "RuntimeError: engine not running." I think the error has something to do with the settings.py file but I'm not sure. Here is the code for the settings.py file. I can also post the main spider if that would help. Any feedback is appreciated:

    BOT_NAME = "testscraper"

    SPIDER_MODULES = ["testscraper.spiders"]
    NEWSPIDER_MODULE = "testscraper.spiders"
    #Scrapy-playwright
    DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    }
    TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

    # Crawl responsibly by identifying yourself (and your website) on the user-agent
    USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"

    # Obey robots.txt rules
    ROBOTSTXT_OBEY = True
    FEED_EXPORT_ENCODING = "utf-8"


    Here is the Spider itself

    import scrapy
    from testscraper.items import NesoItem
    from scrapy.loader import ItemLoader
    from urllib.parse import urljoin

    class BeginnerSpider(scrapy.Spider):
    name="Beginner"
    start_urls = ["https://www.suruga-ya.com/en/category/5"]

    def link_explorer(self,response):
    price_links=response.xpath("//div[@class='cate_body']//a[contains(@href,'price')]/@href").getall()
    print(f"[THE PRICE LINKS ARE {price_links}]")
    for link in price_links:
    full_link=urljoin(response.url,link)
    print(f"[YIELDING REQUEST FOR {full_link}]")
    yield
    scrapy.Request(url=f"{full_link}?page=1",meta={"playwright":True},callback=self.parse)


    def parse(self, response):
    print("[PARSE]")

    nesos=response.css(".product_wrap")

    if not nesos:
    print("[NO ITEMS FOUND]")
    yield from self.link_explorer(response)
    return

    for neso in nesos:

    loader=ItemLoader(item=NesoItem(),selector=neso) #We can add a selector up here so we no longer have to call neso.css, saving us some keystrokes.
    loader.add_css('Title',"h3.group.title_product a::text")
    loader.add_css('Price',".price-new::text") #when a value is added here, the input processor and output processor are used

    print(["yielding"])

    yield loader.load_item()

    next_button=response.xpath("//li/a[contains(text(),'Next')]")
    if next_button:
    next_page=next_button.xpath('@href').get()
    if next_page:
    next_page=urljoin(response.url,next_page)

    yield scrapy.Request(url=next_page,callback=self.parse)

    Continue reading...

Compartilhe esta Página