[Python] Testing multiple pyspark pipelines with different environment variables loaded into a...

Ok so I have a question- For context: I am a new QA working in a team that uses pyspark as our ETL engine. I use pytest as my testing framework.

We have multiple pipelines (ETL’S, I’m gonna use this term from now on) in the same repo. of course, we use environment variables when running the pipeline (we inject them through airflow). The problem being - the env variables are loaded into python variables, to a “RUN ENV” file, which is a global file for all pipelines. Why is this a problem?

When I test the whole repo through CI, the first pipeline test that runs initializes the environment variables (through conftest.py), and they are in turn loaded into RUN ENV.

This causes an annoying situation When a pipeline finishes running, and a new one begins: the RUN ENV has already been set up with the initial env variables I configured, and now it should be configured differently for the next pipeline! I can go ahead and reconfigure each and every variable in RUN ENV (according to the current conftest being loaded) before every e2e test- but it seems wrong. I know the use of RUN ENV is legitimate: usually only one pipeline is being invoked from a repo. But how can I work around this in a neat way in order to make everything clean and reasonable, so that RUN ENV could be reconfigured every other test? Or maybe defined differently so that this issue won’t matter?

I tried to decouple a lot of the code from the run env file even though I shouldn’t really touch pure source code Tried to put env in fixtures rather than just into the conftest but it really doesn’t solve the problem as long as run env is loaded too soon

Continue reading...