[Python] What is the proper way to process None values in Scrapy?

I'm using Scrapy v2.12.0 and Item Loader. My spider returns None for some item fields in certain items. For instance, the field 'exterior_color' is processed as follows::

in the spider, in parse_myitem(self, response) method:

l = ItemLoader(item=MyItem(), response=response)
l.add_xpath('exterior_color', "//working XPath")


in items.py:

class MyItem(scrapy.Item):
exterior_color = scrapy.Field(
input_processor=MapCompose(handle_empty_color, process_color),
output_processor=TakeFirst())


in items.py:

def handle_empty_color(input_list):
return input_list if input_list else ['other']


def process_color(text):
color = text.lower()
match color:
case 'beige':
return 'beige'
case _:
return 'other'


in pipeline.py:

exterior_color = adapter.get('exterior_color')


which is later saved to a relational database using SQLAlchemy, where in models.py,

exterior_color: Mapped[Optional[str]] = mapped_column(String(10))


The intended logic is as follows: handle_empty_color() should return ['other'] if it receives None. Subsequently, process_color() is supposed to handle the case where the input is 'other'. However, I've discovered that the database (as seen in pgAdmin4) contains [none] as double-clicking the field shows it as empty.

My investigation using DevTools and the Scrapy shell indicates that this occurs when add_xpath returns None. While this seems like the correct way to handle missing data, it's not working as expected in my scenario.


I have implemented a workaround, which I understand is not the recommended practice in Scrapy. In the spider, I'm using the 'fuel' field as another instance:

fuel = response.xpath("//working XPath").get()
fuel = "Other" if fuel == None else fuel

l.add_value('fuel', fuel)


What is the proper way to process None values in Scrapy?

Continue reading...