[Python] How to find words in the digits of π?

I have this file that contains the first million decimal places of π Pillion.txt.

I want to find "words" in the digits of π, more specifically I want to convert it to a byte sequence, and find all consecutive runs of word characters. A "word" here means a single string that contains only characters in the union of UNICODE category "L" and 0123456789, or in Python: all(c.isalpha() or c.isdigit() for c in s).

I used two methods to convert the digits to bytes, first I get rid of the decimal point, one method is just using a regex to find the longest runs of decimal digits that can be valid bytes, the other one is converting the digits to an integer and then to bytes.

I then just loop through the bytes character by character and check if the character is a valid word character. If the character is a valid character, append to chunk, else collect chunk and reset chunk.

import re
import sys
from collections import defaultdict
from typing import List

sys.set_int_max_str_digits(1048576)
with open("D:/Pillion.txt", "r") as f:
Pillion = f.read()


BYTES = re.compile(r"1\d\d|2[0-4]\d|25[0-5]|[1-9]\d?|0")


def get_bytes(s: str) -> List[int]:
return list(map(int, BYTES.findall(s)))


PI_BYTES = bytes(get_bytes(f"3{Pillion[2:]}"))

def find_words(byte_seq: bytes) -> defaultdict

• :
  words = defaultdict(list)
  chunk = ""
  for i, b in enumerate(byte_seq):
  if (c := chr(b)).isalpha() or c.isdigit():
  if not chunk:
  index = i
  
  chunk += c
  else:
  if chunk:
  words[chunk].append(index)
  
  chunk = ""
  
  return words
  
  words = dict(sorted(find_words(PI_BYTES).items(), key=lambda x: len(x[0])))
  Pint = int(f'3{Pillion[2:]}')
  Pint_length = Pint.bit_length()
  PI_BYTES1 = Pint.to_bytes((Pint_length >> 3) + Pint_length % 8, byteorder='big')
  words1 = dict(sorted(find_words(PI_BYTES1).items(), key=lambda x: len(x[0])))
  
  
  It works, but I won't show you the output, because most of the "words" are nonsensical. I haven't devised a way to find pronounceable words yet.
  
  Now I want to find all valid UNICODE word characters encoded in UTF8 in π, and of course I cannot just do PI_BYTES.decode('utf8') because that will raise UnicodeDecodeError. So far my method only works with single byte characters, but most UNICODE characters are multi-byte, which aren't supported by my method.
  
  Is there a way to upgrade this to UNICODE?
  
  Continue reading...