[Python] How to scrape PDF files from the list of websites and subdomains without hitting...

Question:


I need to collect links to PDF files (course syllabi) from university websites. I have a large list of base domains like:

• https://university-a.com
• http://www.institute-b.edu
• https://edu-c.example.org

However, the files are often hosted on subdomains — for example:

• docs.university-a.com;
• cabinet.institute-b.edu;
• info.edu-c.example.org

This causes problems with search coverage, especially since I don't have a complete list of subdomains.

What I’ve tried:

Searching via Bing and DuckDuckGo


I formed queries like:

query <- paste0("filetypedf course_name site:", domain)
search_url <- paste0("https://www.bing.com/search?q=", URLencode(query))


In the browser, I often see the desired PDF link in the first result. But in R, I get either a blank page or a result from an unrelated domain.

Parsing search results using rvest


resp <- GET(search_url, user_agent("Mozilla/5.0"))
page <- content(resp, as = "parsed")
links <- page %>% html_nodes("a") %>% html_attr("href")
pdf_links <- links[grepl("\\.pdf", links, ignore.case = TRUE)]


Sometimes I get results, but often not a single PDF link — even if I can clearly see one in the browser.

Handling subdomains

I tried removing https:// and www. to make the site: filter broader:

main_domain <- gsub("^(https?://)?(www\\.)?", "", site)


This helped in some cases but wasn't reliable — Bing and DuckDuckGo still often failed to return results from subdomains.

CAPTCHA & Anti-bot protection


I added random delays and custom User-Agent headers:

Sys.sleep(sample(8:15, 1))
GET(url, user_agent("Mozilla/5.0"))


But DuckDuckGo still shows CAPTCHA after 15–20 requests. Bing often returns empty or incomplete result pages.

What I want to understand:


Is there a reliable way to search and extract PDF links from domains and their subdomains:

• without hitting CAPTCHA, or
• by solving CAPTCHAs automatically, or
• using search engines at all?

Any solutions in R or Python are welcome.

Continue reading...