I've been scouring the internet trying to find a solution to this, and I have yet to find an answer despite this seeming like something that should be a very common use case: How can I safely type check methods or arguments in Python across different files or packages without creating a circular dependency? Let me give an example of something that should work, and does work in most other strongly typed language (eg C/C++, Java, and even Hack), but does not seem to have a sane solution in Python. Let's take the example of a very common data structure - a Graph. I want to organize the structure into multiple files to separate concerns. In duck-typed, Python, it might look like this: # Node.py class Node: def __init__(self): self.edges = [] # Will be a list of Edge objects def add_edge(self, edge): self.edges.append(edge) ... # Edge.py class Edge: def __init__(self, node, weight): self.node = node self.weight = weight Now in the above example, neither object needs to import the other, so everything works. At runtime, another class such as Graph might be responsible for creating nodes and edges, and adding the edges to the nodes. No circular dependency so far. But suppose I want to implement typechecking to ensure no one accidentally attempts to pass an integer type to add_edge, or a string to "Edge()". I would type it like the following: # Node.py from graph.Edge import Edge class Node: def __init__(self): self.edges: List[Edge] = [] # Will be a list of Edge objects def add_edge(self, edge: Edge): self.edges.append(edge) ... # Edge.py from graph.Node import Node class Edge: def __init__(self, node: Node, weight: int): self.node: Node = node self.weight: int = weight This of course presents a problem, because we now have a circular dependency. In our attempt to make the code more type-safe, we've actually introduced something that will crash at runtime. The most common solution I've seen posted for this is to use if TYPE_CHECKING. But this will cause the typechecker to miss a very common, and critical case - where someone tries to use the import for something other than type hinting. Let's take an example: # Node.py if TYPE_CHECKING: from graph.Edge import Edge class Node: def __init__(self): self.edges: List[Edge] = [] # Will be a list of Edge objects def add_edge(self, edge: Edge): self.edges.append(edge) This prevents the circular import, and running pyright or pyre will show no errors, since the typing is correct. But now suppose someone adds a new method to the Node class like the following: # Node.py if TYPE_CHECKING: from graph.Edge import Edge class Node: def __init__(self): self.edges: List[Edge] = [] # Will be a list of Edge objects def add_edge(self, edge: Edge): self.edges.append(edge) def easy_add_edge(self, node: "Node", weight: int): self.add_edges(Edge(node, weight)) The developer then runs pyright, which confirms that there are no type errors in this method. They push it to production, and suddenly everything breaks. Since the import for Edge was inside if TYPE_CHECKING which is False at runtime, python will fail to resolve self.add_edges(Edge(node, weight)) and throw an exception. But the typechecker didn't pick up on this because when it ran, TYPE_CHECKING was True! You might be thinking "why didn't the developer write tests for that method", and you'd be right, but this is exactly the type of bug that typechecking is supposed to solve for! For most developers coming from other languages, the typechecker would be the test for that type of bug! Without knowing a ton about the inner workings of the python typehinting system, the solution would normally seem obvious - typehint using fully qualified names and never go near the TYPE_CHECKING variable (we could even write a lint rule to catch cases where people are doing this). But this is the part of the problem where I've hit a wall, as I can't figure out if this is something that is, or ever will be supported in Python. So the question: How can I safely typecheck code in python in a way that both does not introduce circular dependencies, and also ensures developers aren't trying to use undeclared types at runtime? Can fully-qualified names be used, and if so how? Note: It is not reasonable to place everything that may need a type hinted dependency in the same file, or even the same package. Separation of concerns into multiple packages is something that is expected on any engineering team for almost all production sized applications. I know there is a way to do this if everything is in the same file, but there's no way I can convert my 200k line Django application into a flat-structured single file app. Continue reading...
