I am trying to build a vector database of open-source literature. First step would be extracting the actual units of text from Wikisource. The XML dump seemed the perfect place to go to, but I am having a hard time figuring out how to do that. Pages with namespace 0 are supposed to be the actual articles, but many of those do not contain the actual narrative texts, and are generally a mess. This is probably a better question for someone familiar with the XML structure of the dump, but I am new to programming and might be missing something here. But surely there must be an easy way to get and parse the core content of Wikisource? Continue reading...
