I wonder how I can filter out objects from struct by condition that their id is in list presented in other column. Suppose I have dataframe df with columns event which has the type of list of Struct with fields id and time. There is another column target_ids which contains lists of id. I want to filter from each row of column events only those events, which have id in corresponding list of target_ids. If target_ids is a column of single id of integer type, than the following works: df = df.withColumn('target_time', F.expr('filter(events, x -> x.id == target_ids)').getField('time') However if I change target_ids to list and == to in then I get a syntax error. How can I solve this? And is there any documentation about syntax of expressions inside expr? This page is sadly blank Continue reading...
