I am new to Fabric and have been doing a few uDemy courses however I am not sure how to approach this problem. I have been running a pipeline on my Fabric Workspace for a while now that uses a spark notebook to read, transform and write csv files from my lake house into delta tables in my Silver lake house. The transformations are small, renaming columns and replacing some blank values with NULL. I am told that for smaller batch jobs, such as appending 1 or 2 records per day. Using spark is inefficient use due to the use of several nodes and could instead be done just by using a single node Python notebook with Pandas/Polars. It does seem like Delta was build off of Spark so using it to read and write into Delta tables makes the most sense. But a lot of people suggest not using it for simpler jobs. Is it possible to just connect to the lake house SQL endpoint, write a SQL query using SQLAlchemy and then write that query into the lake house DB using the endpoint? Or is there another way? Or do you recommend to just use Spark for all processing jobs, whether they are small or big. Thank you. Continue reading...
