[Python] Query on snowflake sparks connector return error

I attempt to perform a query by using snowflake sparks, the code looks like this:

command = f'''insert into {NewTable} ({Newcolumn})
select {column} from {Table} {Where}'''
print(command)
df = spark.read.format(snowflake_source_name) \
.options(**snowflake_options) \
.option("query", command) \
.load()


the print result looks like this:

insert into Process.dim_ICON (ICON,dim_ICON_Key)
select ICON, ROW_NUMBER() OVER (ORDER BY ICON) from APIDATA


however when I run the code it return error:

2024-12-01 16:36:44 ERROR error:An error occurred while calling o302.load.
: net.snowflake.client.jdbc.SnowflakeSQLException: SQL compilation error:
syntax error line 1 at position 15 unexpected 'insert'.
syntax error line 1 at position 62 unexpected ')'.


where this error will not occur when it is running in Snowflake SQL

The script should have perform query

insert into Process.dim_ICON (ICON,dim_ICON_Key)
select ICON, ROW_NUMBER() OVER (ORDER BY ICON) from APIDATA


by using pysparks connector

Continue reading...