[Python] Repository foundry palantir pyspark, save code .py

I have a repository pyspark in palantir foundry, I would save when I schedule the run of this, all the code used. I used this code in a transform, for now non-incremental, to launch with all the others and that I thought saved me all the code.

from pathlib import path
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType


spark = SparkSession.builder.getOrCreate()


cartella_repository = Path("/path/to/folder/of/repository")


data = []

for filepath in cartella_repository.rglob("*.py"):
try:
code = filepath.read_text(encoding="utf-8")
data.append((str(filepath.relative_to(cartella_repository))), code)
except Exception as and:
print(f"Error reading {filepath}: {and}")


scheme = StructType([
StructField("filepath", StringType(), True),
StructField("code", StringType(), True)
])


df_spark = spark.createDataFrame(data=data, schema=schema)




It should be very simple, but when I write the dataframe in the output, this is empty. It doesn’t find any file. py imagine, the path I insert is where the repository plus its name. I also tried to add Folder/repositoryname/src

Nothing, always empty table. Where could the error be? Thank you for your help

Continue reading...