I have been assigned to debug a long-running python script in prod. The problem here is that there is no print or debug commands in the script. Wanted to check if there's internal logging that python maintains where it keeps track of which command is being currently run. e.g. My current script below. Here the location has huge files and the process is running for each file. This script is running for the last 14 hours and I am unable to find which command is currently running. So any internal python log where the currently running command is maintained will be of help here. I just need help with the log file directory, or how to find such a log file directory. ... # Read data for fname in glob('<location>*'): df = pd.read_csv(fname,header=None,sep=',') #here needs to modify the trigger feature every time df.columns = [colnames] df = df[df.cps_count>0].replace(r'\s+',np.nan,regex=True).replace('\\N',np.nan) # df = df[df.cps_count>0].replace(r'\s+',np.nan,regex=True).replace('\\N',np.nan) df = pd.get_dummies(df,columns=['prim_ppt']).fillna(np.nan) cols_obj = df.columns[df.dtypes.eq('object')] df[cols_obj] = df[cols_obj].apply(pd.to_numeric, errors='coerce') #this = pd.concat([ids.reset_index()for i in np.setdiff1d(cols,df.columns):df = 0,pd.DataFrame(scores)],axis=1) xref_ids = df['cust_xref_id'] for i in np.setdiff1d(cols,df.columns):df = 0 #xre_id needs to be replaced #feature_importance also needs to be replaced feature_importance = model.predict(xgb.DMatrix(df[[i for i in cols]],df['res'],missing=np.nan),pred_contribs=True) combined=np.c_[feature_importance,xref_ids] df_result=pd.DataFrame(combined,columns=cols+['bias_term','cust_xref_id']) dfs.append(df_result) final= pd.concat(dfs,axis=0) #need to adjust for every model df_result2=final[[colnames]] #need to adjust for every model df_rank=df_result2[[<somecolnames>]].rank(axis=1,method='first',numeric_only=None,na_option='keep',ascending=False,pct=False) df_rank['cust_xref_id']=df_result2['cust_xref_id'] #drop the null cust_xref_id df_rank=df_rank[df_rank['cust_xref_id'].notnull()] df_rank['cust_xref_id']=df_rank['cust_xref_id'].astype('int') #Data transformation #transform from wide to long type df_final=pd.melt(df_rank,id_vars=['cust_xref_id'],value_vars=[<somecolnames>]) df_final = df_final.sort_values(by=["cust_xref_id", "variable"]) #Note: output file has to be tab seperated -- according to Sahil from CSP core team if not os.path.exists('<outputPath>'): os.makedirs('<outputPath') df_final.to_csv('<outputfile>',index=None,header=None,sep='\t') Continue reading...
