[Python] requests.get() still hangs despite timeout – how to force hard cutoff on slow/broken...

I'm working on a script that queries dataset URLs (CSV, JSON, PDF, HTML, sometimes XML) from the GovData.de SPARQL endpoint.

The script loops over many dataset entries and uses requests.get(url, timeout=10) to fetch previews of those files (e.g. first 5 lines for CSVs).

However, despite the timeout, the script still hangs occasionally – no exceptions, no output, and no progress.

I suspect some endpoints (especially .xml, .gml, or WMS/WFS services) respond improperly or stall indefinitely. I need a hard cutoff – if no response within e.g. 15 seconds, the script must continue without blocking.

Here's a simplified version of what I'm using:


import requests
from bs4 import BeautifulSoup
from PyPDF2 import PdfReader
from io import BytesIO
import json

def fetch_preview(url: str, max_lines: int = 5) -> str:
"""
Fetches a short preview of a dataset from various formats.
Hangs for some URLs – suspecting request timeout isn't enforced?
"""
if "service=wms" in url.lower() or url.lower().endswith((".wms",)):
return "(Preview not available for WMS services)"

try:
# This sometimes hangs indefinitely – even with timeout?
resp = requests.get(url, timeout=10)
resp.raise_for_status()
except Exception as e:
return f"(Error loading: {e})"

ctype = resp.headers.get("Content-Type", "").lower()

if "csv" in ctype or url.lower().endswith(".csv"):
return "\n".join(resp.text.splitlines()[:max_lines])

if "application/json" in ctype or url.lower().endswith(".json"):
try:
data = resp.json()
if isinstance(data, list):
snippet = data[:max_lines]
else:
keys = list(data.keys())[:max_lines]
snippet = {k: data[k] for k in keys}
return json.dumps(snippet, ensure_ascii=False, indent=2)
except:
return "(Invalid JSON)"

if "html" in ctype or url.lower().endswith((".html", ".htm")):
soup = BeautifulSoup(resp.text, "html.parser")
paras = soup.find_all("p", limit=max_lines)
return "\n".join(p.get_text().strip() for p in paras)

if "xml" in ctype or url.lower().endswith((".xml", ".gml")) or "wfs" in url.lower():
return "\n".join(resp.text.splitlines()[:max_lines])

if "pdf" in ctype or url.lower().endswith(".pdf"):
try:
reader = PdfReader(BytesIO(resp.content))
if reader.pages:
text = reader.pages[0].extract_text() or ""
return "\n".join(text.splitlines()[:max_lines])
except:
return "(PDF could not be read)"

return "(Unknown format)"

Continue reading...