[Python] markdownify does not produce good markdown - includes characters like \n, \xa0

I'm looking to convert an html string to a markdown string for storage & retrieval of documents in a RAG solution.

I've been trying with beautifulsoup and markdownify packages and finding they don't really do what I'd expect them to do.

For example: My input is something like:

<h3>Section Title</h3><p>Section content. Text text text</p><p>Paragraph 2:</p><ul><li>list item</li><li>list item</li><li>list item</li><li>list item</li></ul><p>Paragraph 3: (1) list item (2) list item (3) list item (4) list item (5) list item



Here I left the formatting but obscured the text. You can see it has html tags like <h3>, <p>, <li>, and numbers in the format (1).

Markdownify produces: (reproducible by calling md(string))

'### Section Title\n\nSection content. Text text text\n\nParagraph 2:\n\n* list item\n* list item\n* list item\n* list item\n\nParagraph 3: (1\\) list item (2\\) list item (3\\) list item (4\\) list item (5\\) list item\n\n'


It's replacing a lot of these html tags with escape characters, newline characters, and other junk that doesn't actually work in markdown!

What's the point? And is there a better way to do this?

Continue reading...