[Python] How to scrape webpage with scrapy framework?

I am newbie in webscraping. I have started learning scrapy framework.

I covered basic tutorial of scrapy. Now I am trying to scrap this page.

As per this tutorial, to get whole html-page contain one should write following code:

import scrapy


class ClothesSpider(scrapy.Spider):
name = "clothes"

start_urls = [
'https://www.chumbak.com/women-apparel/GY1/c/',
]

def parse(self, response):
filename = 'clothes.html'
with open(filename, 'wb') as f:
f.write(response.body)


This code run fine. But I am not getting expected result.

When I open clothes.html, the html-data is not same as when I do inspect from browser. Lot of things are missing in clothes.html.

I didn't understand what's going wrong here. Please help me to push forward. Any help would be appreciated.

Thanks.

Continue reading...