[Python] How to get all Unicode ranges a font has glyphs for and the exceptions?

I am trying to write a Python program to convert Markdown as used by Stack Exchange directly into images. The logic of Markdown is quite simple, I find MathJax to be challenging to render, but I have found this.

But I have trouble determine which font to use for a given character. I want to have total control on which individual character is rendered using which font. More specifically I like this Andika font. It is Sans-Serif, and most Sans-Serif fonts are guilty of not distinguishing UPPER-CASE i with lower-case L, in Andika there are two bars on UPPER-CASE i, and its lower-case a has no "hair" on its head...

And it supports Greek and Cyrillic scripts too, but sadly it doesn't support Hiragana, Katakana, Hangul, Chinese etc. I will use the script to "serialize" text from many different languages, predominantly the text is in English, but it frequently contains whole lyrics of songs in other languages. I plan to use Andika for European languages and Noto Sans SC for East Asian script, but neither of them supports emojis, for that I need yet another font.

I need to way to determine if a chosen font has glyph for a given character, this has to be done for every single character, a character will be first checked against Andika to check if it can be rendered using Andika, with fallbacks to Noto Sans SC and then Noto Color Emoji, and if all three fail an exception needs to be raised which will be handled...

For English text this is simple enough, just ord(c) < 128, but to efficiently check if a given font supports a UNICODE codepoint is hard.

I have decided on the following method, first, the notation (a, b) specifies a range of integers, a >= 0 and b >= a, and both ends are included, for example, (0, 0) is a valid range containing only 0, and of course (5, 9) contains (5, 6, 7, 8, 9).

Then the task is to find all consecutive ranges in a font, and do binary search for x in the list of starts, and then check if x is inside the corresponding range.

I did with the following (you need fontTools and unicodedata2):

import unicodedata2
from bisect import bisect_right
from fontTools.ttLib import TTFont

def get_glyphs_in_font(font_path):
return sorted(
{code for cmap in TTFont(font_path)["cmap"].tables for code in cmap.cmap}
)


def group_consecutive(lst):
result = []
start = end = -1e309
for i in lst:
if i == end + 1:
end = i
else:
if end != -1e309:
result.append((start, end))

start = end = i

result.append((start, end))
return result


def get_glyph_ranges(font_path):
return group_consecutive(get_glyphs_in_font(font_path))


Its output is correct, but it has 129 elements in the output for "Andika-Regular.ttf". It would be inefficient to do this for every single character.

Also, UNICODE has a bunch of control characters, reserved characters, blank characters etc. they are all non-printable so I call all of them invalid, and the following finds all of them:

INVALID_UNICODE = [
i
for i in range(1114112)
if unicodedata2.category(chr(i)) in {"Cc", "Cf", "Cn", "Co", "Cs", "Zl", "Zp", "Zs"}
]

INVALID_UNICODE_SET = set(INVALID_UNICODE)


Now let me explain the rules: let (a, b) (c, d) be two ranges, d >= c > b >= a, now if any of the following is a subset of INVALID_UNICODE: (b, c), (b + 1, c), (b + 1, c - 1), (b, c - 1), then the two ranges should combine to form (a, d). Now if the above doesn't apply, but c / b <= 1.05, then they should still combine, but now all valid codepoints in the range (b + 1, c - 1) should be accreted as exceptions.

For example, given the first 10 elements of get_glyphs_in_font("Andika-Regular.ttf"):

[(32, 126),
(160, 328),
(330, 800),
(803, 831),
(838, 879),
(903, 903),
(915, 916),
(920, 920),
(926, 926),
(928, 928),
]


The first is adjacent to the range (0, 32) so the first iteration should have (0, 126), after that, (127, 160) are all invalid, so it should combine the second range to form (0, 328). Then there is a discontinuity between (0, 328) and (330, 800), but according to the second rule they should be joined and [329] be noted as an exception.

The outcome for the example should be:

((0, 928),
[329,
801,
802,
832,
833,
834,
835,
836,
837,
880,
881,
882,
883,
884,
885,
886,
887,
890,
891,
892,
893,
894,
895,
900,
901,
902,
904,
905,
906,
908,
910,
911,
912,
913,
914,
917,
918,
919,
921,
922,
923,
924,
925,
927])


I had tried this half a dozen times but couldn't get the intended result. This is my latest attempt, it is both smart and stupid at the same time:

def get_glyph_groups1(font_path):
ranges = group_consecutive(get_glyphs_in_font(font_path))
result = {}
cur_start, cur_end = 0, 32
cur_missing = []
for start, end in ranges:
s1 = cur_end + cur_end not in INVALID_UNICODE_SET
e1 = start - start not in INVALID_UNICODE_SET
i1 = bisect_right(INVALID_UNICODE, e1)
i2 = bisect_right(INVALID_UNICODE, s1)

if i1 - i2 == e1 - s1 and bool(i1 - i2) == bool(e1 - s1):
cur_end = end
continue

if start / cur_end <= 1.05:
cur_missing.extend(range(cur_end + 1, start))
cur_end = end
else:
result[(cur_start, cur_end)] = cur_missing
cur_start, cur_end = start, end
cur_missing = []

result[(cur_start, cur_end)] = cur_missing
return result


Needless to say it doesn't work:

In [36]: get_glyph_groups1("C:/Windows/Fonts/Andika-Regular.ttf")
Out[36]: {(0, 122654): []}


How can I make it work?


I have changed my code to this:

INVALID_UNICODE = [
i
for i in range(1114112)
if unicodedata2.category(chr(i)) in {"Cc", "Cf", "Cn", "Co", "Cs", "Zl", "Zp", "Zs"}
]

INVALID_UNICODE_RANGES = group_consecutive(INVALID_UNICODE)
INVALID_UNICODE_STARTS = [start for start, _ in INVALID_UNICODE_RANGES]
INVALID_UNICODE_RANGES = dict(INVALID_UNICODE_RANGES)

def should_combine(a, b):
i = bisect_right(INVALID_UNICODE_STARTS, a)
start = INVALID_UNICODE_STARTS[max(i - 1, 0)]
end = INVALID_UNICODE_RANGES[start]
if not start <= a <= end:
i = bisect_right(INVALID_UNICODE_STARTS, a + 1)
start = INVALID_UNICODE_STARTS[max(i - 1, 0)]
end = INVALID_UNICODE_RANGES[start]

if not start <= a <= end:
return False

return start <= b - 1 <= end if not start <= b <= end else True


def get_glyph_groups3(font_path):
ranges = group_consecutive(get_glyphs_in_font(font_path))
result = {}
cur_start, cur_end = 0, 32
cur_missing = []
for start, end in ranges:
if should_combine(cur_end, start):
cur_end = end
continue

if start / cur_end <= 1.05:
cur_missing.extend(range(cur_end + 1, start))
cur_end = end
else:
result[(cur_start, cur_end)] = cur_missing
cur_start, cur_end = start, end
cur_missing = []

result[(cur_start, cur_end)] = cur_missing
return result


But it still doesn't work:

In [56]: get_glyph_groups3("C:/Windows/Fonts/Andika-Regular.ttf").keys()
Out[56]: dict_keys([(0, 126), (160, 1327), (6832, 6842), (7424, 10217), (11360, 11835), (42752, 43877), (61744, 67726), (119820, 122654)])


The first one should be (0, 1327). The output is closer to what I wanted, but sadly I cannot verify the correctness of the output.

Continue reading...