[Python] Django string disappears in template when applying Unicode filter

I have a self-written tool with is converting Indian transliterated strings into Unicode output, working like this:

>>> x_HK_to_UNI("ahaM")


Which results in the following string:

'aha\xe1\xb9\x82''


So far so good, this tool is tested and has been used on converting thousand of documents. Now I am writing a Django app and wish to apply this function as a filter in a template. The code looks like this:

from __future__ import unicode_literals
from django import template
from django.utils.encoding import smart_text
from xliterator import *
register = template.Library()

@register.filter
def process_trans_for_html(string):
newstring = []
string = string.split(' ')
for word in string:
if word[-3:] == '_xn' :
newstring.append(x_HK_to_UNI(word[:1]).upper()+(x_HK_to_UNI(word[1:-3]).lower())
elif word[-3:] == '_xh' :
newstring.append(x_HK_to_UNI(word[:-3]).lower())
else:
newstring.append(word)
return ' '.join(newstring)


In the template I pass data to the filter this way:

{{ line.line|process_trans_for_html }}


Basically the function is working as long as nothing is replaced, so when it processes yena_xn it is displayed on the website correctly as Yena. But ahaM_xn should be turned into ahaṃ (this is the job of x_HK_to_UNI) - and when such a word is encountered, the whole string just disappears and leaves blank space on the page.

Any suggestion on this? I tried already all kinds of Unicode transformation stuff and there seems to be no much help. I feel rather confused. There is also no error message.

I use Python2.7.

EDIT: Here the relevant parts of xliterator.py:

import sys, re, os

SEPARATOR_PRIMARY=";"

class Xlator(dict): # Xlator is initialized through a mapping
def _make_regex(self):
"""Build re object based on the keys of the current dict"""
return re.compile("|".join(map(re.escape, self.keys())))
def __call__(self, match):
"""Handler invoked for each regex match"""
return self[match.group(0)]
def xlate(self, text):
"""Translate text, returns the modified text"""
return self._make_regex().sub(self, text)

def _transposeDict(d): # returns a dict with key value transposed
nd = {}
for (k,v) in d.items():
nd[v]=k
return nd

#Unicode to HK
DICT_UNI_HK = {'ā':'A','Ā':'A','ī':'I','Ī':'I','ū':'U','Ū':'U',\
'ṛ':'R','Ṛ':'R','ṝ':'RR','ḷ':'L','Ḷ':'L','ḹ':'LL',\
'ṃ':'M','Ṃ':'M','ḥ':'H','Ḥ':'H','ṅ':'G','Ṅ':'G',\
'ñ':'J','Ñ':'J','ṭ':'T','Ṭ':'T','ḍ':'D','Ḍ':'D',\
'ṇ':'N','Ṇ':'N','ś':'z','Ś':'z','ṣ':'S','Ṣ':'S',
}

def x_UNI_to_HK(in_str_UNI): # input:
xlator_Obj = Xlator(DICT_UNI_HK)
return xlator_Obj.xlate(in_str_UNI)

def x_HK_to_UNI(in_str_UNI):
DICT_HK_UNI = dict (zip(DICT_UNI_HK.values(),DICT_UNI_HK.keys()))
xlator_Obj = Xlator(DICT_HK_UNI)
return xlator_Obj.xlate(in_str_UNI)

Continue reading...