[Python] Trouble Scraping Codeur.com — Are JavaScript or Anti-Bot Measures Blocking My Script?

I’ve been trying to scrape the project listings from Codeur.com using Python, but I'm hitting a wall — I just can’t seem to extract the project links or titles.

Here’s what I’m after: links like this one (with the title inside):

Acquisition de leads

Pretty straightforward, right? But nothing I try seems to work.

So what’s going on? At this point, I have a few theories:

JavaScript rendering: maybe the content is injected after the page loads, and I'm not waiting long enough or triggering the right actions.

Bot protection: maybe the site is hiding parts of the page if it suspects you're a bot (headless browser, no mouse movement, etc.).

Something Colab-related: could running this from Google Colab be causing issues with rendering or network behavior?

Missing headers/cookies: maybe there’s some session or token-based check that I’m not replicating properly.

What I’d love help with Has anyone successfully scraped Codeur.com before?

Is there an API or some network request I can replicate instead of going through the DOM?

Would using Playwright or requests-html help in this case?

Any idea how to figure out if the content is blocked by JavaScript or hidden because of bot detection?

If you have any tips, or even just want to quickly try scraping the page and see what you get, I’d really appreciate it.

What I’ve tested so far

1. requests + BeautifulSoup I used the usual combo, along with a user-agent header to mimic a browser. I get a 200 OK response and the HTML seems to load fine. But when I try to select the links:

soup.select('a[href^="/projects/"]')

I either get zero results or just a few irrelevant ones. The HTML I see in response.text even includes the structure I want… it’s just not extractable via BeautifulSoup.

1. Selenium (in Google Colab) I figured JavaScript might be involved, so I switched to Selenium with headless Chrome. Same result: the page loads, but the links I need just aren’t there in the DOM when I inspect it with Selenium.

Even something like:

driver.find_elements(By.CSS_SELECTOR, 'a[href^="/projects/"]')

returns nothing useful.

Continue reading...