[Python] How can I pass is_split_into_words option to LayoutLMv3Processor?

I'm fine-tuning a LayoutLMv3 model using HuggingFace Transformers. During preprocessing, I want to use is_split_into_words=True to ensure proper label alignment for token classification.

My setup:

• I'm using LayoutLMv3Processor
• I have a list of words (example["words"]), bounding boxes, and labels
• I want to pass is_split_into_words=True during tokenization

However, calling the processor like this:

LayoutLMv3Tokenizer.from_pretrained("microsoft/layoutlmv3-base")
processor = LayoutLMv3Processor.from_pretrained("microsoft/layoutlmv3-base")

...
def preprocess(example):

image = Image.open(example["image_path"]).convert("RGB")
image_width, image_height = image.size
normalized_bboxes = [normalize_bbox(bbox, image_width, image_height) for bbox in example["bboxes"]]

encoding = processor.tokenizer(
image,
example["words"],
is_split_into_words=True,
boxes=normalized_bboxes,
word_labels=[label2id[l] for l in example["labels"]],
truncation=True,
padding="max_length",
return_tensors="pt"
)

return {
"input_ids": encoding["input_ids"].squeeze(0),
"attention_mask": encoding["attention_mask"].squeeze(0),
"bbox": encoding["bbox"].squeeze(0),
"pixel_values": encoding["pixel_values"].squeeze(0),
"labels": encoding["labels"].squeeze(0)
}


tokenized_dataset = dataset.map(preprocess, remove_columns=dataset.column_names)


But it throws:

TypeError: LayoutLMv3TokenizerFast._batch_encode_plus() got an unexpected keyword argument 'is_split_into_words'

It looks like LayoutLMv3Processor.__call__() doesn’t forward is_split_into_words to the tokenizer.

I would appreciate any help!

Continue reading...