Using Python, I want to read text from a (utf-8 encoded) text file, but at the same time need to know the start and end position of each character in the file (in bytes). As there might be multi-byte characters, this isn't a 1:1 mapping. I can see that reading through the file character by character and keeping an offset akin to offset += len(c.encode('utf-8')) (or whatever the encoding of the file would be), but this seems a bit ad-hoc, especially once (ignored) decoding errors etc. come into play. Is there a standard way / library to do that? I would imagine a list of character-offset pairs or a str plus a list of integers which contains the offsets. Continue reading...
