[Python] Python If Statement with multiple "and" conditions, if not default column value

Python 3.10.12 using DataBricks. I am trying to get these filter results:

example:

If "column_1" = '2024' and column_2 in ('DE','SC')
then "value_1" else "value_2"

If "column_1" = '2023' and column_2 in ('DE','SC')
then "value_3" else "value_4"

if the record does not fit the criteria above then return the value in column_3.


list1 = (['DE','SC'])


This what I have tried with no success:

# Sample data
data = [
("2024", "DE", "original1"),
("2024", "SC", "original2"),
("2024", "FR", "original3"),
("2023", "DE", "original4"),
("2023", "SC", "original5"),
("2023", "IT", "original6"),
("2022", "DE", "original7"),
("2025", "NL", "original8"),
]

table2 = (table1.withColumn(
'column_3',
when(
(col('column_2').isin(list1)) &
(col('column_1') == ('2024')),
'value_1'
).otherwise(col('value_2'))).otherwise(column_3)


Basically I am trying to add multiple filters to column_3 and if the criteria is not met, return the values in column_3. I do no mind using alternative solutions.

Basically I am getting results with no filtered data.

Continue reading...