[Python] How can I classify sitemap URLs (WooCommerce / OpenCart / custom ) [closed]

I'm working on a crawler in Python that takes an e-commerce sitemap and classifies each URL into a page type, for example:

• home
• product
• product_category
• product_tag
• brand
• post
• privacy_policy
• terms
• cookie_policy
• etc.

The final goal is to have all pages with:

1. Their markdown content (HTML → Markdown)
2. A type label describing what kind of page it is

For the HTML → Markdown conversion I’m using Crawl4AI, which works great.
The difficult part is the classification of each URL into the correct page type.

What I’ve tried so far


For some URLs the pattern is straightforward:

• /product/... → product
• /product_cat/... or /category/... → product_category
• /product_tag/... → product_tag
• /brand/... → brand
• URLs containing privacy, terms, cookie → legal pages

This works fine for platforms like WooCommerce or OpenCart when they use predictable URL patterns.

The main problem


Many shops use SEO-friendly product URLs that don’t follow any consistent pattern.
Example: https://www.example.com/health-aid-vitamin-c-1000mg-orange-eff-20s-pr-vitamin-c1000mg-doro

This is a product page, but:

• there’s no /product/ segment
• there’s no identifiable route like route=product/product
• the slug alone is not reliable without adding many fragile rules

So URL-based classification becomes unreliable.

What I’m looking for


I would really appreciate ideas on how to make the classification more robust, such as:

1. Better product detection signals from HTML, for example:

• application/ld+json with @type: Product
• presence of price, availability, SKU, attributes
• “add to cart” / “buy now” buttons
• common WooCommerce / OpenCart microdata patterns

2. How to structure a maintainable classifier, e.g.:

• a rule-based system with ordered heuristics
• a feature-extraction approach + small ML model (Naive Bayes, Logistic Regression, etc.)
• a hybrid approach combining URL signals + HTML signals

3. Platform-specific hints

• selectors, CSS classes, or schema typically used by WooCommerce / OpenCart / custom themes
• patterns that distinguish category/brand/tag listing pages

Summary


Given only a sitemap (or sitemap index) and the HTML for each URL,
what are reliable ways to classify a page into types like:

product, product_category, product_tag, brand, post, static_page, privacy_policy, terms, cookie_policy, etc.?

I’m primarily interested in:

• robust product detection when SEO URLs don’t follow obvious patterns, and
• architecture advice for building a maintainable classifier in Python.

Continue reading...