I'm working in a kedro project where I want to automatically label thousands of audio files, apply transformations to them and then store them in a folder of folders, each subfolder corresponding to one label. I want that folder of folders to be a catalog entry on my yml file I followed this Kedro tutorial and created my own custom dataset for saving/loading .wav files in kedro catalog. I also am able to create PartitionedDataset catalog entries in catalog.yml such as audio_folder: type: partitions.PartitionedDataset dataset: my_kedro_project.datasets.audio_dataset.SoundDataset path: data/output/audios/ filename_suffix: ".WAV" for saving/loading folders of .WAV files in Kedro catalog. The next level of abstraction I would require is to be able to create a catalog entry corresponding to a folder containig folders such as the audio_folder above. I don't want to do this by dynamicaly creating catalog entries but by extending PartitionedDataset class. This is because I want the folder of folders to be part of my catalog.yml. My questions are Is this possible? Has any of you ever try something like this? In case it is possible, my custom class should only include the _load, _save, and _describe methods as when I'm customizing an AbstractDataset? Thank you for your help Continue reading...
