[Python] Why does "SettingWithCopyWarning" get raised (seemingly) inconsistently?

Setup


We first import pandas and numpy and then create a 5000 rows x 10 columns pd.DataFrame called df

import pandas as pd
import numpy as np

# Generate non-trivial df
data = np.random.rand(5000, 10)
column_names = [f'col_{i}' for i in range(10)]
df = pd.DataFrame(data, columns=column_names)

Get SettingWithCopyWarning warning


We then take df and select 2 columns and add to them a new column with the static value 10

foo = df
print(f"foo has type {type(foo)}\n")
bar = foo[["col_0", "col_1"]]
bar["new"] = 10
print(f"After modification bar has {type(bar)}\n\nbar is:\n{bar}")


This gives the output

foo has type <class 'pandas.core.frame.DataFrame'>

After modification bar has <class 'pandas.core.frame.DataFrame'>

bar is:
col_0 col_1 new
0 0.837911 0.715060 10
... ... ... ...
4999 0.684007 0.176949 10

[5000 rows x 3 columns]
C:\Users\vents\AppData\Local\Temp\ipykernel_5824\1295636698.py:5: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
bar["new"] = 10


Notice the warning at the bottom

Do the same to a different df, but don't get SettingWithCopyWarning


Now we do the same as above, but set foo to a trivial table

foo = pd.DataFrame([["x",20.2,30], ["y",80,70]], columns=["col_0", "col_1", "col_2"])
print(f"foo has type {type(foo)}\n")
bar = foo[["col_0", "col_1"]]
bar["new"] = 10
print(f"After modification bar has {type(bar)}\n\nbar is:\n{bar}")


This now gives the output

foo has type <class 'pandas.core.frame.DataFrame'>

After modification bar has <class 'pandas.core.frame.DataFrame'>

bar is:
col_0 col_1 new
0 x 20.2 10
1 y 80.0 10


We have no warning.

My explanation(?) and thoughts


As far as I understand, SettingWithCopyWarning means that there is uncertainty around whether the table is a copy or a view. So maybe pandas has some optimization logic where if the table is small it just gives you a copy because, "oh what the hell, just save them the headache; their table is tiny, we'll just make a copy for them"??? But then for a bigger table it doesn't want to waste RAM making a copy so it requires users to be smarter?

That's my best guess.

I'm just not happy with taking the generic advice of, "Oh, just use foo.loc[:, ["col_0", "col_1"] or foo[["col_0", "col_1"]].copy() and you won't get the error", because I am not sure I fully understand the problem...

Like what other parts of my code are bad, but I'm getting (un)lucky to not getting the warning?

From what I've seen online most SettingWithCopyWarnings often get thrown if you take a subset of your data that has a different type to your original table and then try to do stuff to it (e.g. foo[0]["col_0"], which first takes the series foo[0] and then from that gives you the column col_0).

Continue reading...