[Python] Why is scrapy's @id selector not working?

Here is the page I'm testing on: https://ai.fmcsa.dot.gov/SMS/Carrier/1031421/BASIC/UnsafeDriving.aspx

I'm trying to get this table

<table class="scrollable smsEvents public" id="inspectionTable" data-page="1" data-row-count="4" data-url="/SMS/Carrier/1031421/BASIC/UnsafeDriving/Inspections/--MODE--/--SORTBY--/--PAGE--.aspx?UserType=Public" data-loaded="0">


using this:

inspections = response.xpath('//table[@id="inspectionTable"]')


Scrapy seems to be completely ignoring the @id selector. When I print inspections it's just the first table on the page which doesn't even have an id attribute.

I've printed response.text and confirmed it does contain #inspectionTable so it's not an issue of it being loaded by javascript.

Continue reading...