[Python] HLTV/results scraper not working. Multiple samed named divs

I am building a script to scrape the hltv.org/results page for cs2 matches. However, I am encountering a lot of issues with this, specifically, the website hltv.org/results?offset={} has multiple divs called results-sublist where the results are stored inside in divs called result-con. My script only scrapes the first results-sublist and is unable to scrape any further, resulting in a lot of missing matches.

My function is defiend as

import urllib.request
from urllib.error import URLError, HTTPError
from urllib.request import Request, urlopen # Import Request and urlopen
import bs4
import time
import pandas as pd
from tqdm import tqdm
import random

# Base URL for HLTV results with pagination
base_url = 'https://www.hltv.org/results?offset={}'

def scrape_match_links(base_url, num_pages):
"""
Scrapes match links from the HLTV results pages.

Parameters:
base_url (str): The base URL for the HLTV results page.
num_pages (int): The number of pages to scrape (should be 100 res pr. page).

Returns:
list: A list of match links.
"""
offset = 0
match_links = []
headers = {'User-Agent': 'Mozilla/5.0'}

while offset <= num_pages * 100:
url = base_url.format(offset)
req = urllib.request.Request(url, headers=headers)

try:
with urllib.request.urlopen(req) as response:
webpage = response.read().decode('utf-8')
soup = bs4.BeautifulSoup(webpage, 'html.parser')

# Find all results-sublist divs
results_sublists = soup.find_all('div', class_='results-sublist')
if not results_sublists:
print("No results-sublist found on this page.")
break # Stop if no results are found on this page

for sublist in results_sublists:
results = sublist.find_all('div', class_='result-con')
for result in results:
match_link = result.find('a')
if match_link:
full_link = 'https://www.hltv.org' + match_link['href']
match_links.append(full_link) # Append full URL

offset += 100

except HTTPError as e:
print(f'HTTPError: {e.code} - {e.reason}')
break # Stop if there's an HTTP error (e.g., too many requests)
except URLError as e:
print(f'URLError: {e.reason}')
break
except Exception as e:
print(f'Unexpected error: {str(e)}')
break

return match_links


Any insight into why I might be encountering this and/or idea to work around this is greatly appreciated. Best, Jonas

I have tried to use selenium, it worked on the first page yielding 6 divs of class results-sublist, however, on subsequent pages it returned 0 instances of results-sublist.

Continue reading...