[Python] My __len__() doesn't work correctly with len() and list() for custom Class

The following is a simplified version of what I am trying to do (the actual implementation has a number of nuances):

class SideDict(MutableMapping, dict): # isinstance(obj, (MutableMapping, dict)) should return True
"""
The purpose of this special dict is to side-attach another dict. A key
and its value from main dict are preferred over same key in the
side-dict. If only a key is not present in main dict, then it is used
from the side-dict.
"""

# The starting SideDict instance will have side_dict=None, a subsequent
# SideDict instance can use the first instance as its side_dict.
def __init__(self, data, side_dict: SideDict | None):
self._store = dict(data)
self._side_dict = side_dict
# Also other stuff

# Also implements __bool__, __contains__, __delitem__, __eq__, __getitem__,
# __missing__, __or__, __setitem__ and others.

def __iter__(self):
self._iter_keys_seen = []
self._iter = self._store.__iter__()
return self

def __next__(self):
# (Just a summary of the code as it is somewhat complex and large.
# The logic seems to be working fine.)
# - First run self._iterator.__next__(). All keys are appended to
# self._iter_keys_seen.
# - The moment StopIteration exception is raised, silence it and
# change iterator to self._side_dict.__iter__().
# - If a key in this iterator is already in self._iter_keys_seen (i.e.
# in self._store), then skip and go to next key.
# - Else return it and also add it to self._iter_keys_seen.
# - If StopIteration is raised now, don't silence it.

def __len__(self):
return len([k for k in self]) # Its not the most efficient, but
# I don't know any other way.

sd_0 = SideDict(data=..., side_dict=None)
sd_1 = SideDict(data=..., side_dict=sd_0)
sd_2 = SideDict(data=..., side_dict=sd_1)

print(len(sd_0), len(sd_1), len(sd_2)) # all work fine
print(list(sd_0)) # ! Here is the problem, shows empty list `[]` !


On putting some print()s, here is what I observed being called:

1. list() triggers obj.__iter__() first.
2. Followed by obj.__len__(). I vaguely understand that this is done so as to allocate optimal length of list.
3. Because obj.__len__() has list-comprehension ([k for k in self]), it again triggers obj.__iter__().
4. Followed by obj.__next__() multiple times as it iterates through obj._store and obj._side_dict.
5. When obj.__next__() hits the final un-silenced StopIteration, list-comprehension in obj.__len__() ends.
6. Here the problem starts. list() seems to be calling obj.__next__() again immediately after ending obj.__len__(), and it hits StopIteration again. There is no obj.__iter__(). And so the final result is an empty list!

What I think might be happening is that list() starts an iterator on its argument, but before doing anything else, it wants to find out the length. My __len__() uses an iterator itself, so it seems the both are using the same iterator. And then this iterator is consumed in obj.__len__(), and nothing left for outer list() to consume. Please correct me if I am wrong.

So how can I change my obj.__len__() to use a non-clashing iterator?

Edit: Solution

Based on answer by @user2357112, here is the working code:

class SideDict(...):
def __iter__(self):
return SideDictIterator(self)

# Removed __next__(...), all other stuff remains the same

class SideDictIterator:
def __init__(self, side_dict: SideDict):
self._side_dict = side_dict

self._iter_keys_seen = []
self._iter = self._side_dict._store.__iter__()

def __iter__(self):
return self

def __next__(self):
# Exactly the same stuff that was in SideDict.__next__(),
# except using self._side_dict instead of self

Continue reading...