[Python] How to run custom regression in bigquery dataframes/bigframes (using remote functions)

I am unsure on how to run a custom regression (using the pyfixest package) with several columns using bigquery dataframes and append the regression residual as a new column. I can't figure out how to pass the full dataframe to a remote functions. A previous answer suggested using bpd.Series but this does not work. Furthermore, the pyfixest function requires a pandas dataframe as input.

I am using the following remote function. This function converts the bpd.Series into a bpd.DataFrame and then into a pandas df.

@bpd.remote_function(bpd.Series, int, reuse=False, packages=["pyfixest","pandas"])
def get_residuals(input_df):
import pyfixest as pf
import pandas as pd

input_df = input_df.to_frame().to_pandas()

m1 = pf.feols("spotify_streams ~ song_age_days | isrc", data=input_df)
return m1.resid()

#return input_df["spotify_streams"]


I am then applying this function to the original dataframe (which has columns 'isrc', 'spotify_streams', 'song_age_days')

df = bpd.read_gbq(sql, use_cache = False) #Get the data here, works fine.

df["residual"] = df[
["spotify_streams","song_age_days","isrc"]
].apply(get_residuals, axis=1)


But then it fails with

NameError: `song_age_days` is not present in the dataset or evaluation context.


I have also tried using just input_df = input_df.to_pandas() but then it fails with

File \"<ipython-input-62-316cc1e96dfb>\", line 9, in get_residuals\n File \"/layers/google.python.pip/pip/lib/python3.10/site-packages/pandas/core/generic.py\", line 6204, in __getattr__\n return object.__getattribute__(self, name)\nAttributeError: 'Series' object has no attribute 'to_pandas'\n"}
.


even though bpd.Series should have a .to_pandas() command.

Continue reading...