Consider the model that is trained in this exemple at Kedro's documentation from typing import Any, Dict from kedro.pipeline import node, pipeline from pyspark.ml.classification import RandomForestClassifier from pyspark.sql import DataFrame def train_model(training_data: DataFrame) -> RandomForestClassifier: """Node for training a random forest model to classify the data.""" classifier = RandomForestClassifier(numTrees=10) return classifier.fit(training_data) def predict(model: RandomForestClassifier, testing_data: DataFrame) -> DataFrame: """Node for making predictions given a pre-trained model and a testing dataset.""" predictions = model.transform(testing_data) return predictions def create_pipeline(**kwargs) -> Pipeline: return pipeline( [ node(train_model, inputs=["training_data"], outputs="example_classifier"), node( predict, inputs=dict(model="example_classifier", testing_data="testing_data"), outputs="example_predictions", ), ] ) I would like to not loose my pre-trained model and save it as a Pyspark's PipelineModel but I could not find a suitable dataset for that in Kedro dataset's documentation Usually I would do something like this: save_path = "path/to/save/pipeline_model" pipeline_model.save(save_path) But as I'm using Kedro, I don't want to have IO outside of my catalog. Is this a supported use case, or would I have to implement my own data custom KedroDataset to achieve this? Continue reading...
