[Python] Capturing Wikipedia Topic Hierarchy

I want scale my current project to use Wikipedia article linking, and Wikipedia library in Python, to capture a hierarchical mathematical subject dependency and display it in a tree structure.

For now, I have prepared a program for my university course offerings where the subject (or course) dependencies are explicitly given such that I can create the following JSON of courses based on their pre/parallel requisites (one or two way subject dependency):

"Analysis 2": {
"parallel_requisites": ["Analysis 1"],
"pre_requisites": [],
},
"Linear Algebra 2": {
"parallel_requisites": ["Linear Algebra 1"],
"pre_requisites": [],
},
"Topology": {
"parallel_requisites": ["Analysis 2", "Linear Algebra 2"],
"pre_requisites": [],
}


With my program, given the JSON structure above, I can produce the following tree:



Where straight lines are pre-requisites and dashed are parallel-requisites in the figure (the displaying and manipulation of the tree I have ready).

I want to have this JSON structure but on a grander scale using Wikipedia. A good place to start, I figured, is the article for fields of math. From it, my program can make its way down such that for example from the article "Algebra” we go to "Linear Algebra". As I tell my program to go down in this linking chain, I have two questions which I am not really sure how to answer:

1. When do I tell the program to stop? When is the article sufficient enough in depth but not too spesific such that I capture general fields?
2. How can I capture dependency of subjects on one another?

For inspiration, I found this math roadmap. For now, I just want to capture the topic hierarchy automatically from Wikipedia without the other features.

Although for now the generality of the article I hope to end is rather vague, a good first description is an article which has enough depth such that a book is published about it or a course is taught on it.

I found this page to crawl but still some of the linked articles there aren't fitting to what I defined above and the question of understanding one and two way dependencies between subjects remains.

Could you please suggest clever ways to capture hierarchy and subject detection using the Wikipedia Python package?

Continue reading...