I want to parse sitemap and find out all urls from sitemap and then appending some word to all urls and then I want to check response code of all modified urls. for this task I decided to use scrapy because it have luxury to crawl sitemaps. its given in scarpy's documentation with the help of this documentation I created my spider. but I want to change urls before sending for fetching. so for this I tried to take help from this link. this link suggested my to use rules and implement process_requests(). but I am not able to make use of these. I tired little bit that I have commented. could anyone help me write exact code for commented lines or any other ways to do this task in scrapy? from scrapy.contrib.spiders import SitemapSpider class MySpider(SitemapSpider): sitemap_urls = ['http://www.example.com/sitemap.xml'] #sitemap_rules = [some_rules, process_request='process_request')] #def process_request(self, request, spider): # modified_url=orginal_url_from_sitemap + 'myword' # return request.replace(url = modified_url) def parse(self, response): print response.status, response.url Continue reading...
