[Python] Getting Error Code: ScriptExecution.StreamAccess.NotFound error while trying to read...

I am trying to read in a MLTABLE in my pipeline, but getting:

Error Code: ScriptExecution.StreamAccess.NotFound
Native Error: error in streaming from input data sources
StreamError(NotFound)
=> stream not found
NotFound
Error Message: The requested stream was not found. Please make sure the request uri is correct.| session_id=1f8669ce-5a60-494b-a8dd-fd07fee8b186


When I try to read in the mltable in an interactive session, it works fine:

import mltable
tbl = mltable.load(f'azureml:/{flight_data.id}')
tbl.to_pandas_dataframe()


The above code works fine.

I am reading in the data using:

flight_data = ml_client.data.get(name='flightdelaydata1', version='2')


Below is the .py code:

%%writefile {data_prep_folder}/data_prep.py
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import mlflow
import mltable
import os

import argparse

def main():

#Main function of the file

parser = argparse.ArgumentParser()
parser.add_argument('--data', help='Input data for flight delay model', type=str)
parser.add_argument('--train_test_split_ratio', help='Test data proportion', type=float, default=.20)
parser.add_argument('--train_data', help='Training Data', type=str)
parser.add_argument('--test_data', help='Test Data', type=str)

args = parser.parse_args()

# Start logging

mlflow.start_run()

tbl = mltable.load(args.data)
df = tbl.to_pandas_dataframe()
mlflow.log_metric('Number of observations', df.shape[0])
mlflow.log_metric('Number of features', df.shape[1])

df = df.dropna()
df = df.loc[ : , ['Month', 'DayofMonth', 'DayOfWeek', 'DepDelay', 'DepDel15', 'ArrDel15', 'Cancelled', 'ArrDelay']]

Train_data, Test_data = train_test_split(df, test_size=args.train_test_split_ratio)

Train_data.to_csv(os.path.join(args.train_data, 'train.csv'))
Test_data.to_csv(os.path.join(args.test_data, 'test.csv'))

mlflow.end_run()


if __name__ == "__main__":
main()


and the command code is as below:

data_prep_component = command(name='flight_delay_model_data_prep',
description='Flight Delay Model Prediction Data Preparation Component',
display_name='Flight Delay Data Prep',
inputs = {
'data' : Input(type='mltable', path = flight_data.id),
'train_test_split_ratio' : Input(type='number')
},
outputs = {
'train_data' : Output(type = 'uri_folder'),
'test_data' : Output(type = 'uri_folder')
},
command= '''python {data_prep_folder}/data_prep.py \
--data ${{inputs.data}} --train_test_split_ratio ${{inputs.train_test_split_ratio}} \
--train_data ${{outputs.train_data}} --test_data ${{outputs.test_data}}''',
environment = f'{envt.name}:{envt.version}'
)


Not sure if am making an error while reading the mltable in the job code. Can someone please help me where I am going wrong.

Continue reading...