[Python] How do I instantiate a dbt NodeSelector object using the dbt core Python API?

I wanna build a Python application that given a path to a dbt project and a dbt selector string (<my_selector>), returns the fully qualified names of the tables and views that would be queried when running dbt run --select <my_selector>. In essence, I want to know which tables and views in my database act as sources to the models selected using <my_selector>. I believe the NodeSelector class from the dbt core Python API can help me here, but I'm having some issues when trying to use this class. This class' constructor expects me to provide a dbt Graph (dbt.graph.graph.Graph) and a dbt Manifest (dbt.contracts.graph.manifest.Manifest), but when I do it and run my code, I get a TypeError: 'dict' object is not callable error.

From what I've been investigating, the error seems to be raised by line 53 of the core/dbt/graph/selector.py file, where the .nodes() function is called. Python seems to resolve nodes as a dict object and those are not callable, so I get this TypeError whenever I try to instantiate an object from this class.

Here's a minimal reproducible example (assumes the existence of a dbt project that was successfully run at some point - manifest.json file exists):

import json
import os

from dbt.contracts.graph.manifest import Manifest
from dbt.graph.graph import Graph
from dbt.graph.selector import NodeSelector

dbt_project_dir = "<path/to/your/dbt/project/>"


manifest_path = os.path.join(dbt_project_dir, "target", "manifest.json")
with open(manifest_path, "r") as f:
manifest_json = json.load(f)
manifest = Manifest.from_dict(manifest_json)
my_graph = Graph(manifest)

# This line errors with a message saying: TypeError: 'dict' object is not callable
NodeSelector(graph=my_graph, manifest=manifest)


The funny this is that if instead of instantiating a NodeSelector object, we run these lines:


try:
print(type(my_graph.nodes))
print(my_graph.nodes())
except Exception as exc:
print(exc)


We get this as output:

<class 'method'>
'dict' object is not callable


This is pretty weird, since the type() function resolves my_graph.nodes as a method, but when I try to actually call it, it's resolved as a dict.

Any ideas on what's happening here? I think it might have something to do with the nodes property of networkx's Digraph, which can return a dict-like object or a method, but I'm not sure. Can someone please help me instantiate this class?

Thanks!

Continue reading...