[Python] SitemapLoader(sitemap_url).load() hangs

from langchain_community.document_loaders import SitemapLoader

def crawl(self):
print("Starting crawler...")

sitemap_url = "https://gringo.co.il/sitemap.xml"
print(f"[CRAWLER] Loading sitemap: {sitemap_url}")
try:
loader = SitemapLoader(sitemap_url)
print("[CRAWLER] SitemapLoader initialized.")
print("[CRAWLER] Calling loader.load()...")
documents = loader.load()
print(f"[CRAWLER] Loaded {len(documents)} documents via LangChain.")
except Exception as e:
print(f"[WARN] SitemapLoader failed: {e}")


This runs prints

print("[CRAWLER] Calling loader.load()...")


and hangs there.


I have no idea why it wouldn't just finish parsing, the sitemap has ~1500 pages.

Continue reading...