[Python] Scraping an Online Food Website: Spider opens but does not begin crawl

This website is structured as a sitemap index with compressed XMLs that I want to grab, decompress, and iterate through their inner URLs to then pull item name and item price.

The problem is I run scrapy crawl spider_name and it opens the spider but does not begin the crawl.

The command scrapy shell "myurl.xml" also does not work, it gets stuck. The XML pages don't have javascript so I don't think that's the issue. The page works in my browser so I am thinking the servers are blocking the request? There is nothing in robots.txt that explicitly disallows scraping but they might have anti bot mechanisms in place.

What could cause this behavior, and how can I troubleshoot or fix it? Are there other headers or configurations I might be missing in Scrapy? Is this a lost cause?

Here's some code:

I have tried adding User-Agent headers, changing ROBOTSTXT_OBEY = False, COOKIES ENABLED = True, and I've tried rotating proxies but it still does not seem to fetch the page and start crawling. It's possible I'm not doing the user-agent headers right or the proxies (which i got by downloading a random list online here: https://github.com/TheSpeedX/PROXY-List/blob/master/http.txt)

import scrapy
from scrapy.spiders import SitemapSpider


class FoodSpider(SitemapSpider):
name = "spider_name"
allowed_domains = ["myurl_index.xml"]

# List the sitemap URLs here
sitemap_urls = [
"myurl_index.xml"
]
custom_settings = {
"USER_AGENT": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
"DOWNLOAD_DELAY": 2, # Delay 2 seconds between requests
"CONCURRENT_REQUESTS_PER_DOMAIN": 2,
}


def parse_product(self, response):
"""
Parse individual product pages and extract desired data.
"""
self.logger.info(f"Scraping product page: {response.url}")
product_name = response.xpath("//h1/text()").get() or "N/A"
product_price = response.xpath("//span[@class='price']/text()").get() or "N/A"
yield {
"url": response.url,
"name": product_name.strip() if product_name else "N/A",
"price": product_price.strip() if product_price else "N/A",
}


settings.py:

```
# Configure proxy middleware
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 1,
'scrapy_proxies.RandomProxy': 100,
}

PROXY_LIST = '/Users/myname/Documents/FreshDirectSpider/food/food/resources/proxies_with_http.txt' # Path to proxy list file
PROXY_MODE = 0 # Use random proxy from the list

```

Continue reading...