[Python] Snowpark error unless cache_result() is used

I have a relatively long pipeline written in Snowpark and now I started to get this error:

SnowparkSQLException: (1304): 01be98bf-0002-488e-0000-32893416fcfe: 000603 (XX000): SQL execution internal error: Processing aborted due to error 300010:391167117; incident 5224277.

I tried to remove steps I added since last succesful testing, but could not identify the culprit. Though, I noticed if I add df = df.cache_result() before the steps that error out - the code is working as expected. So adding a few caching steps helps, but I'd like to understand the root cause.

Some context that might be helpful:

1. I have ~15 input tables. The largest is less than 2M rows.
2. in my code style I reuse variable names. E.g. contract_df = contract_df.join(customer_df..). So the core dataframe name is reused more than 20 times.
3. Warehouse size: XS.
4. Error manifests when I do df.show()
5. Transformations I use: joins, window functions, filtering, adding columns

Continue reading...