[Python] Crawl4AI token threshold not applied to raw html in arun

Here’s a brief overview of what I want to achieve

1. Extract raw htmls and save them
2. Use Crawl4AI to produce a ‘cleaner’ and smaller HTML that has a lot of information, including what I will eventually extract, but also a lot of redundant stuff.
3. Then, extract the relevant information from the intermediate HTML.

However, there’s a crucial issue in step 2.

Problem: the config arguments chunk_token_threshold and apply_chunking set in LLMExtractionStrategy, and which are passed to arun, are not working.

The raw html is not cut into chunks of length specified by chunk_token_threshold. In fact, it is passed ‘as is’. This in turn results in an error because the context window token size is surpassed.

I've identified the following errors and explanations:

• Problem 1: arun triggers aprocess_html in the stack. And there you have these lines of codes ( source )

# # Use IdentityChunking for HTML input, otherwise use provided chunking strategy
chunking = (IdentityChunking() if content_format in ["html", "cleaned_html"]
else config.chunking_strategy
)
sections = chunking.chunk(content)
extracted_content = config.extraction_strategy.run(url, sections)



And IdentityChunking is a "Chunking strategy that returns the input text as a single chunk."

I believe this is the cause of the error. The length of `sections' is one and the content is equal to the raw html.

So then, I tried solving this bug with chunking = config.chunking_strategy

Now, sections is broken down into many chunks as we want. However, a new error came up which I've been unable to solve:

• Problem 2: config.extraction_strategy.run(url, sections) applies another merging procedure to the sections (source same as before). Following this procedure, the sections are again grouped into a single raw HTML. NB: I've tried it with different htmls and the length of the output is not always one - it can be larger, but crucially, the chunks are very large and do not respect chunk_token_threshold)

Help would be much appreciated!!

Sample code

import asyncio
from crawl4ai import (
AsyncWebCrawler,
LLMExtractionStrategy,
LLMConfig,
CrawlerRunConfig,
CacheMode

)
import os
import openai

########### SETUP
llm_prompt_path = "{INSERT}"
html_path = "{INSERT}"

deepseek_key = os.getenv("OPENAI_API_KEY")

# Read prompt instructions
with open(llm_prompt_path, "r", encoding = 'utf-8') as f:
llm_content_parse_instructions : str = f.read()


llm_config = LLMConfig(
provider = "deepseek/deepseek-chat", # Name of the LLM provider
api_token = deepseek_key, # API token for authentication,
temprature = 0.7,
base_url = "https://api.deepseek.com",
)

llm_extraction_strategy = LLMExtractionStrategy(
llm_config = llm_config,
extraction_type="block", # Type of extraction to perform
instruction = llm_content_parse_instructions, # Instructions for the LLM
chunk_token_threshold = 4000, # maximum length of tokens passed to LLM API (if you go above 64k, will throw error)
apply_chunking = True,
input_format = "html", # Format of the input content
verbose=True, # Enable verbose logging
)

# 2. Build the crawler config
run_config = CrawlerRunConfig(
extraction_strategy=llm_extraction_strategy,
cache_mode=CacheMode.BYPASS
)



with open(html_path, encoding = 'utf-8') as f:
raw_html_content = f.read()

async def main():

async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url = f"raw:{raw_html_content}",
config = run_config
)


asyncio.run(main())

Continue reading...