[Python] scrapy: return list of relative urls where a where a certain word appears

On this website , there are many cards which are accessible in multiple areas of the site. I am attempting to scrape the site to return all of its instances(in the form of its url) based on the title. For example, a card titled: AZ School Safety program, is found at: https://lawforkids.org/officers & https://lawforkids.org/educators . My goal is for scrapy to do this for me instead of manually finding every instance.

import scrapy

class PostsSpider(scrapy.Spider):
name = "card"
start_urls = [ 'https://lawforkids.org/' ]

def parse(self, response):
for card in response.css('h5.card-title::text').re(r'AZ School Safety Program'):
yield { 'url': card.css('base::attr(href)').getall() }


Besides not being confident that url selector is correct, my attempt to crawl results in: PostsSpider.parse callback is not defined. Any insight is greatly appreciated, thanks!

Continue reading...