[Python] Unicode Decoding error when trying to generate pdf with non-ascii characters

I am working with some software that is generating an error when trying to create a pdf from html that contains non-ascii characters. I have created a much simpler program to reproduce the problem and help me understand what is going on.

#!/usr/bin/python
#coding=utf8
from __future__ import unicode_literals
import pdfkit
from pyPdf import PdfFileWriter, PdfFileReader
f = open('test.html','r')
html = f.read()
print html
pdfkit.from_string(html, 'gen.pdf')
f.close()


Running this program results in:

<html>
<body>
<h1>ر</h1>
</body>
</html>

Traceback (most recent call last):
File "./testerror.py", line 10, in <module>
pdfkit.from_string(html, 'gen.pdf')
File "/usr/local/lib/python2.7/dist-packages/pdfkit/api.py", line 72, in from_string
return r.to_pdf(output_path)
File "/usr/local/lib/python2.7/dist-packages/pdfkit/pdfkit.py", line 136, in to_pdf
input = self.source.to_s().encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd8 in position 18: ordinal not in range(128)


I tried adding a replace statement to strip the problem character, but that also resulted in an error:

Traceback (most recent call last):
File "./testerror.py", line 9, in <module>
html = html.replace('ر','-')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd8 in position 18: ordinal not in range(128)


I am afraid I don't understand ascii / utf-8 encoding very well. If anyone could help me understand what is going on here, that would be great! I am not sure if this is a problem in the pdf library, or if this is a result of my ignorance of encodings

Continue reading...