I’m working with stateful streaming processing in PySpark, specifically using applyInPandasWithState, mapGroups… etc. My function processes data maintaining state but also handles timeouts (e.g. GroupStateTimeout.ProcessingTimeTimeout). I want to understand best practices for unit testing (using pytest or unittest) such functions, i.e. mocking Spark/GroupState behaviour completely vs using an actual Spark session and how we would go about testing timeouts in either case. Initially, I decided to mock Spark’s behaviour completely, to have full control over tests. This allowed me to test the outcome data received in a specific orders). However, I am now struggling to mock timeout behaviour properly. I’m unsure whether my current mock-based approach is too far from real production behaviour. Initially, I decided to mock Spark’s behaviour completely, to have full control over tests. This allowed me to test the outcome data received in a specific orders). However, I am now struggling to mock timeout behaviour properly. I’m unsure whether my current mock-based approach is too far from real production behaviour. Continue reading...
