I am new to webscraping and would like to scrape the links from the site below using scrapy: https://shop.coles.com.au/a/national/everything/search/bread?pageNumber=1 I created the below xpath to scrape the links and when I test it out by going to inspect and pressing ctrl + f I get 51 matches which is equal to the number of products and so seems to be correct: //span[@class="product-name"]/../../@href However when I go into scrapy shell with the link and apply the command: response.xpath('//span[@class="product-name"]/../../@href').extract() with or without a User agent I just get an empty list. When I run the shell I get a 429 error, which indicates I have made too many requests. But as far as I am aware I have only made 1 request. In addition I have also set up a spider for this where I set CONCURRENT_REQUESTS = 1 and also get a 429 error. Does anyone know why my xpath doesn't work on this site? Thanks Edit Below is the spider code: import scrapy class ColesSpider(scrapy.Spider): name = 'coles' allowed_domains = ['shop.coles.com.au'] start_urls = ['https://shop.coles.com.au/a/national/everything/search/bread/'] def parse(self, response): prod_urls = response.xpath('//span[@class="product-name"]/../../@href').extract() for prod_url in prod_urls: yield{"Product_URL": prod_url} Continue reading...
