[Python] Scrapy Crawlspider does not work with 507 status code

Discussão em 'Python' iniciado por Stack, Maio 3, 2025.

  1. Stack

    Stack Membro Participativo

    &nbsp &nbsp &nbsp &nbsp I have a scrapy Crawlspider that parses reviews, using a scrapy-rotating-proxies.

    &nbsp &nbsp &nbsp &nbsp But when I tried to connect to the site I got the 507 status code. In spite of that, I managed to connect to the site with the code 200 and get the information for a few times.
    &nbsp &nbsp &nbsp &nbsp As I understood the idea, the status 507 indicates the server does not have enough capacity to process the request. Or was I blocked after all? Is there anything I can do in this situation? I really need to get the data.

    My spider:

    class PagesCrawlSpider(CrawlSpider):
    name = "pages_crawl"

    allowed_domains = ["otzovik.com"]
    start_urls = ["https://otzovik.com/clothes_and_footwear/footwear_male/sports_shoes_for_men/"]

    rules = (Rule(LinkExtractor(restrict_xpaths="//div[@class='caption']/h3/a"), callback="parse_item", follow=True),)

    def parse_item(self, response):
    item = {}
    item["brand"] = response.xpath('//div[@class="page-caption"]/h1[@class="product-name"]/span/span[@class="fn"]/text()').get()
    item["plus"] = response.xpath('//div[@class="review-body-wrap"]/div[@class="review-plus"]/text()').getall()
    item["minus"] = response.xpath('//div[@class="review-body-wrap"]/div[@class="review-minus"]/text()').getall()
    return item


    Middlewares.py

    DOWNLOADER_MIDDLEWARES = {
    "rotating_proxies.middlewares.RotatingProxyMiddleware": 450,
    }
    ROTATING_PROXY_LIST_PATH = "{MY_PATH}/rotating_proxy_list.txt"



    Result

    scrapy crawl pages_crawl -o review_pages.json
    2025-05-03 15:41:06 [scrapy.utils.log] INFO: Scrapy 2.12.0 started (bot: review_pages)
    2025-05-03 15:41:06 [scrapy.utils.log] INFO: Versions: lxml 5.4.0.0, libxml2 2.11.9, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 24.11.0, P
    ython 3.13.0 (tags/v3.13.0:60403a5, Oct 7 2024, 09:38:07) [MSC v.1941 64 bit (AMD64)], pyOpenSSL 25.0.0 (OpenSSL 3.4.1 11 Feb 2025), cryptography 44.0.2, Platform Windows-11-10.0.26100-SP0
    2025-05-03 15:41:06 [scrapy.addons] INFO: Enabled addons:
    []
    2025-05-03 15:41:06 [asyncio] DEBUG: Using selector: SelectSelector
    2025-05-03 15:41:06 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
    2025-05-03 15:41:06 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.windows_events._WindowsSelectorEventLoop
    2025-05-03 15:41:06 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
    2025-05-03 15:41:06 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.windows_events._WindowsSelectorEventLoop
    2025-05-03 15:41:06 [scrapy.extensions.telnet] INFO: Telnet Password: *********
    2025-05-03 15:41:06 [scrapy.middleware] INFO: Enabled extensions:
    ['scrapy.extensions.corestats.CoreStats',
    'scrapy.extensions.telnet.TelnetConsole',
    'scrapy.extensions.feedexport.FeedExporter',
    'scrapy.extensions.logstats.LogStats',
    'scrapy.extensions.throttle.AutoThrottle']
    2025-05-03 15:41:06 [scrapy.crawler] INFO: Overridden settings:
    {'AUTOTHROTTLE_ENABLED': True,
    'AUTOTHROTTLE_MAX_DELAY': 3,
    'AUTOTHROTTLE_START_DELAY': 1,
    'BOT_NAME': 'review_pages',
    'COOKIES_ENABLED': False,
    'DOWNLOAD_DELAY': 3,
    'FEED_EXPORT_ENCODING': 'utf-8',
    'NEWSPIDER_MODULE': 'review_pages.spiders',
    'ROBOTSTXT_OBEY': True,
    'SPIDER_MODULES': ['review_pages.spiders'],
    'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
    2025-05-03 15:41:07 [scrapy.middleware] INFO: Enabled downloader middlewares:
    ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
    'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
    'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
    'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
    'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
    'rotating_proxies.middlewares.RotatingProxyMiddleware',
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
    'scrapy.downloadermiddlewares.retry.RetryMiddleware',
    'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
    'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
    'scrapy.downloadermiddlewares.stats.DownloaderStats']
    2025-05-03 15:41:07 [scrapy.middleware] INFO: Enabled spider middlewares:
    ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
    'scrapy.spidermiddlewares.referer.RefererMiddleware',
    'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
    'scrapy.spidermiddlewares.depth.DepthMiddleware']
    2025-05-03 15:41:07 [scrapy.middleware] INFO: Enabled item pipelines:
    []
    2025-05-03 15:41:07 [scrapy.core.engine] INFO: Spider opened
    2025-05-03 15:41:07 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2025-05-03 15:41:07 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
    2025-05-03 15:41:07 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 0, unchecked: 5, reanimated: 0, mean backoff time: 0s)
    2025-05-03 15:41:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://otzovik.com/robots.txt> (referer: None)
    2025-05-03 15:41:07 [scrapy.core.engine] DEBUG: Crawled (507) <GET https://otzovik.com/clothes_and_footwear/footwear_male/sports_shoes_for_men/> (referer: None)
    2025-05-03 15:41:07 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <507 https://otzovik.com/clothes_and_footwear/footwear_male/sports_shoes_for_men/>: HTTP status code is not handled or not allowed
    2025-05-03 15:41:07 [scrapy.core.engine] INFO: Closing spider (finished)
    2025-05-03 15:41:07 [scrapy.extensions.feedexport] INFO: Stored json feed (0 items) in: review_pages.json
    2025-05-03 15:41:07 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 588,
    'downloader/request_count': 2,
    'downloader/request_method_count/GET': 2,
    'downloader/response_bytes': 3412,
    'downloader/response_count': 2,
    'downloader/response_status_count/200': 1,
    'downloader/response_status_count/507': 1,
    'elapsed_time_seconds': 0.354455,
    'feedexport/success_count/FileFeedStorage': 1,
    'finish_reason': 'finished',
    'finish_time': datetime.datetime(2025, 5, 3, 12, 41, 7, 479104, tzinfo=datetime.timezone.utc),
    'httpcompression/response_bytes': 1078,
    'httpcompression/response_count': 1,
    'httperror/response_ignored_count': 1,
    'httperror/response_ignored_status_count/507': 1,
    'items_per_minute': None,
    'log_count/DEBUG': 7,
    'log_count/INFO': 13,
    'proxies/mean_backoff': 0.0,
    'proxies/reanimated': 0,
    'proxies/unchecked': 5,
    'response_received_count': 2,
    'responses_per_minute': None,
    'robotstxt/request_count': 1,
    'robotstxt/response_count': 1,
    'robotstxt/response_status_count/200': 1,
    'scheduler/dequeued': 1,
    'scheduler/dequeued/memory': 1,
    'scheduler/enqueued': 1,
    'scheduler/enqueued/memory': 1,
    'start_time': datetime.datetime(2025, 5, 3, 12, 41, 7, 124649, tzinfo=datetime.timezone.utc)}
    2025-05-03 15:41:07 [scrapy.core.engine] INFO: Spider closed (finished)

    Continue reading...
    Stack, Maio 3, 2025
    #1

Compartilhe esta Página