I have searched through a large amount of documentation to try to find an example of what I'm trying to do. I admit that the bigger issue may be my lack of python expertise. So i'm reaching out here in hopes that someone can point me in the right direction. I am trying to create a python function that dynamically queries tables based on a function parameters. Here is an example of what i'm trying to do: def validateData(_ses, table_name,sel_col,join_col, data_state, validation_state): sdf_t1 = _ses.table(table_name).select(sel_col).filter(col('state') == data_state) sdf_t2 = _ses.table(table_name).select(sel_col).filter(col('state') == validation_state) df_join = sdf_t1.join(sdf_t2, [sdf_t1 == sdf_t2 for i in join_col],'full') return df_join.to_pandas() This would be called like this: df = validateData(ses,'table_name',[col('c1'),col('c2')],[col('c2'),col('c3')],'AZ','TX') this issue i'm having is with line 5 from the funtion: df_join = sdf_t1.join(sdf_t2, [col(sdf_t1) == col(sdf_t2) for i in join_col],'full') I know that code is incorrect, but I'm hoping it explains what i'm trying to do. If anyone has any advice on if this is possible or how, I would greatly appreciate it. Continue reading...
