[Python] TypeError:Invalid argument, not a string or column

I am trying to do some source to target testing in pyspark. First part I am trying to do is a count of columns using a Lean Six Sigma method making sure there are less than 3/1000000 discrepancies in the columns. When I run this though, the if statement throws a:

TypeError: Invalid argument, not a string or column: -276244 of type <class 'int'>. For columns literals, use 'lit', 'array','struct' or 'create_map' function.
​

Could anyone help?

import pyspark.sql.functions as f
from pyspark.sql.types import *

good_fields = []
bad_fields = {}
count_issues = {}


columns = list(spark.sql('show columns from tu_historical').toPandas()['col_name'])

for col in columns:
print(col)
df = spark.sql(f'select pid,fnum,{col} from historical_clean')
df1 = spark.sql(f'select pid,fnum,{col} from historical1')

#count issue testing
if abs(df1.count()-df.count()) > df1.count()*.000003:
count_issues[col] = df1.count()-df.count()

test_df = df.join(df1,(df.num == df1.file) & (df1.pid == df.pid),'left').filter(df1[col]!=df[col])

Continue reading...