[Python] Python: Running tests with all combinations of feature flags

We have several modules that require mandatory feature / backout flags. These flags are defined at module level.

module.py:

from enabled import is_enabled

FLAGS = {flag : is_enabled(flag) for flag in ("foo", "bar")}

if FLAGS.get("foo"):
def baz():
print("New baz")
else:
def baz():
print("Old baz")

print(f"Loaded module with {FLAGS=}")
print("#"*100)


enabled.py:

def is_enabled(_):
return True


I can use mock.patch to patch is_enabled and by using importlib.reload I can test all combinations of True and False for all flags for a module(max number of flags at the same time is 5 or 6 so combination is not going to explode).

My main problem is now running the test with all combinations; I have tried:

1. 
   Creating a decorator : this fails because setup and teardown methods are not called between repeated runs in the decorator causing assertions to fail. I can either call self.setUp and self.tearDown in each test OR pass those methods to the decorator and call them but it feels like a messy approach which creates dependencies and is fragile. I'm going to avoid this if possible.
   
   
2. 
   Creating a test runner (subclass of TextTestRunner) : Unfortunately a test runner already exists and is setup. I'm not sure if it is possible / how to use my own test runner instead of the existing test runner (which can't be easily swapped out / refactored as it is used a lot). Is there a way for me to override / ignore that and use my own? This approach is cleaner and technically more correct as the the test suite is ran multiple times with all the necessary setup /tear-down, captured results, etc.
   

given the unit test below, can you please demonstrate how to run the test_baz with all combinations of "foo" and "bar" being True and False?

module_test.py

import module
from importlib import reload
from unittest import TestCase
from itertools import product
from functools import wraps, partial
from unittest.mock import patch, DEFAULT


def run_all_combinations_decorator(func):
flags = sorted(module.FLAGS.keys())
combinations = list(product((True, False), repeat=len(flags)))

def is_enabled(new_flags, flag):
return new_flags.get(flag, DEFAULT)

@wraps(func)
def wrapper(*args, **kwargs):
for combination in combinations:
new_flags = {flag: enabled for flag, enabled in zip(flags, combination)}
print(f"Running with {new_flags=}")
with patch("enabled.is_enabled", side_effect=partial(is_enabled, new_flags)):
reload(module)
result = func(*args, **kwargs) # Not ideal as only the last result is captured
return result

return wrapper


class ModuleTests(TestCase):
def setUp(self):
self.x = 0

@run_all_combinations_decorator
def test_baz(self):
# self.setUp() # Uncommenting this makes the test pass but is not ideal
module.baz()
self.x += 1
self.assertEqual(self.x, 1) # Fails because setup is not called between runs


If there is a better approach to this that I have not yet tried, please suggest that as well.

Continue reading...