[Python] Unable to Extract Data from Dynamically Loaded Content Using Splash

I'm trying to scrape funding opportunity data from https://marie-sklodowska-curie-actions.ec.europa.eu/funding. The data I need includes:

• Title of the funding opportunity
• Status (e.g., open, closed)
• Deadline
• Link to the opportunity

The website uses dynamic content loading (likely via JavaScript), and I'm using Scrapy with Splash to render the page. However, my current code isn't extracting any data. Here's the code I'm using:

import scrapy
from notices.items import MarieCurieItem
from scrapy_splash import SplashRequest

class MscaSpider(scrapy.Spider):
name = "msca"

def start_requests(self):
url = 'https://marie-sklodowska-curie-actions.ec.europa.eu/funding'
yield SplashRequest(url=url, callback=self.parse)

def parse(self, response):
msca_item = MarieCurieItem()
for funding in response.css('article.eac-calls-teaser'):
msca_item['title'] = funding.css('h3::text').get()
msca_item['status'] = funding.css('span.deadline-time::text').get()
msca_item['link'] = funding.css('a::attr(href)').get()
yield msca_item


Problem:

1. The response.css selectors aren't matching any elements, even though the data is visible in the browser.
2. When I inspect the page, the funding opportunities are loaded dynamically (likely via an API or JavaScript).

What I've Tried:

1. I used Splash to render the page, but the data still isn't accessible.
2. I checked the Splash logs and confirmed that the page is loading, but the dynamic content isn't being captured.

Questions:

1. How can I modify my Splash script to wait for the dynamic content to load?
2. Are there better selectors I should use to extract the required data?
3. Should I directly target the API endpoint instead of rendering the page with Splash? If so, how can I identify the correct API endpoint?

Additional Information:

• When I inspect the page, the funding opportunities are inside a <eac-faceted-search> component.
• I tried increasing the wait time in Splash, but it didn't help.

Any guidance or suggestions would be greatly appreciated!

Continue reading...