[Python] Removing strange/special characters from outputs llama 3.1 model

Background: I'm using Hugging Face's transformers package and Llama 3.1 8B (Instruct).

Problem: I am generating responses to a prompt one word at a time in the following way (note that I choose over texts and append that to the input_string, then repeat the process):

tokenizer = AutoTokenizer.from_pretrained(model_path, use_safetensors=True)
model = AutoModelForCausalLM.from_pretrained(model_path, use_safetensors=True)

input_ids = tokenizer.encode(input_string, return_tensors="pt") # tokenize to ids
logits = model(input_ids).logits # call model() to get logits
logits = logits[-1, -1] # only care about the last projection in the last batch
probs = torch.nn.functional.softmax(logits, dim=-1) # softmax() to get probabilities
probs, ids = torch.topk(probs, 5) # keep only the top 5
texts = tokenizer.convert_ids_to_tokens(ids) # convert ids to tokens


But I notice many strange or special characters appearing in the output. For example, the following is the literal string returned from input_string = "How often should I wear a seatbelt?":

ĠAlways.ĊĊĊÄÃĦAlways,ĠunlessĠyouĠareÄłinÃĦaÃĦcarÃĥthatÃĦisÃĦnotÃĦmoving.


Is there any way to easily remove strange special characters?

I've tried using options on the decoder (in every possible T/F combo), such as the following:

myStr = 'ĠAlways.ĊĊĊÄÃĦAlways,ĠunlessĠyouĠareÄłinÃĦaÃĦcarÃĥthatÃĦisÃĦnotÃĦmoving.'
tokenizer.decode(tokenizer.encode(myStr), skip_special_tokens=True, clean_up_tokenization_spaces=True)


But it doesn't remove any of the special characters from the string.

Continue reading...