[Python] BeautifulSoup Not Finding Table Headers on ClinicalTrials.gov Despite Inspect Element...

I am very new to Python, and I want to use the Beautifulsoup library to fetch the clinical Trials data ("mitochondrial diseases") for my research studies. Although they have an API, I want to use web scraping.

URL = https://clinicaltrials.gov/search?
cond=mitochondrial%20diseases&page=1


There are 772 studies that I want to fetch (till page number 8).

Although I can see that the headers are in the "th" tag under the inspect element, I am not able to fetch them. Either they show [] or None. I tried different combinations, but I am not getting them all. Further, I want to fetch all the details under those headers for all the studies listed.

I am attaching the screenshots of all.

Please help me with this.

I am writing the code as follows:

from bs4 import BeautifulSoup
import requests
import pandas as pd


url = 'https://clinicaltrials.gov/search?cond=mitochondrial%20diseases&page=1'

response = requests.get(url)

soup = BeautifulSoup(response.content, 'html.parser')

print(soup.find_all("th", class_="ngcontent-ng-c1423484431"))

OutPut: []

Continue reading...