[Python] simple scrapy based on...

My code based on examples that I searched did not seem to function as intended so I decided to use a working model found on github: https://github.com/scrapy/quotesbot/blob/master/quotesbot/spiders/toscrape-xpath.py

I then modified it slightly to showcase what I am running into. The code below works great as intended but my ultimate goal is to pass the scraped data from first "parse" to a second "parse2" function so that I can combine data from 2 different pages. But for now I wanted to start very simple so I can follow what is happening, hence the heavily stripped code below.

# -*- coding: utf-8 -*-
import scrapy
from quotesbot.items import MyItems
from scrapy import Request


class ToScrapeSpiderXPath(scrapy.Spider):
name = 'toscrape-xpath'
start_urls = [
'http://quotes.toscrape.com/',
]

def parse(self, response):
item = MyItems()
for quote in response.xpath('//div[@class="quote"]'):
item['tinfo'] =
quote.xpath('./span[@class="text"]/text()').extract_first()
yield item



but then when I modify the code as below:

# -*- coding: utf-8 -*-
import scrapy
from quotesbot.items import MyItems
from scrapy import Request


class ToScrapeSpiderXPath(scrapy.Spider):
name = 'toscrape-xpath'
start_urls = [
'http://quotes.toscrape.com/',
]

def parse(self, response):
item = MyItems()
for quote in response.xpath('//div[@class="quote"]'):
item['tinfo'] =
quote.xpath('./span[@class="text"]/text()').extract_first()
yield Request("http://quotes.toscrape.com/",
callback=self.parse2, meta={'item':item})

def parse2(self, response):
item = response.meta['item']
yield item


I only have one item scraped and it says the rest are duplicates. It also looks like "parse2" is not even read at all. I have played with the indentation and the brackets thinking I am missing something simple, but without much success. I have looked at many examples to see if I can make sense of what could be the issue but I still am not able to make it work. I am sure its a very simple issue for those gurus out there, so I yelp "Help!" somebody!

also my items.py file looks like below and I think those two files items.py and toscrape-xpath.py are the only ones in action as far as I can tell since I am quite new to all this.

# -*- coding: utf-8 -*-`enter code here`

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class QuotesbotItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
pass

class MyItems(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
tinfo = scrapy.Field()
pass


Thank you very much to any and all help you can provide

# -*- coding: utf-8 -*-
import scrapy
from quotesbot.items import MyItems
from scrapy import Request


class ToScrapeSpiderXPath(scrapy.Spider):
name = 'toscrape-xpath'
start_urls = [
'http://quotes.toscrape.com/',
]

def parse(self, response):
item = MyItems()
for quote in response.xpath('//div[@class="quote"]'):
item =
{'tinfo':quote.xpath('./span[@class="text"]/text()').extract_first()}
**yield response.follow**('http://quotes.toscrape.com', self.parse_2,
meta={'item':item})

def parse_2(self, response):
print "almost there"
item = response.meta['item']
yield item

Continue reading...