[Python] Why does this python script randomly crashes?

I have the following script that randomly crashes when I run it and I cannot figure out why. The script is a smaller version of a larger script, but still reproduces the error of the larger script.

import pandas as pd
import numpy as np

dflength = 7303787

PLUs = pd.DataFrame(np.random.randint(100000,200000,size=(dflength, 1)), columns=['PLU'])
ArticleNames = pd.DataFrame(np.random.randint(1,100,size=(dflength, 1)), columns=['ArticleName'])
df = pd.concat([PLUs, ArticleNames], axis=1)

PLU_List = df['PLU'].unique().tolist()

for iPLU in PLU_List:
print(df[df["PLU"]==iPLU]["ArticleName"].shape)
df3 = df[df["PLU"]==iPLU]["ArticleName"].copy()
print(df3.shape)
print(df3.iloc[-1])


When it crashes it shows an output like this:

49
(82,)
(82,)
85
(59,)
(59,)
93
(62,)
(0,)
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
File \\nas\user\dashboard_23.py:18
16 df3 = df[df["PLU"]==iPLU]["ArticleName"].copy()
17 print(df3.shape)
---> 18 print(df3.iloc[-1])

File ~\miniconda3\envs\main\Lib\site-packages\pandas\core\indexing.py:1191, in _LocationIndexer.__getitem__(self, key)
1189 maybe_callable = com.apply_if_callable(key, self.obj)
1190 maybe_callable = self._check_deprecated_callable_usage(key, maybe_callable)
-> 1191 return self._getitem_axis(maybe_callable, axis=axis)

File ~\miniconda3\envs\main\Lib\site-packages\pandas\core\indexing.py:1752, in _iLocIndexer._getitem_axis(self, key, axis)
1749 raise TypeError("Cannot index by location index with a non-integer key")
1751 # validate the location
-> 1752 self._validate_integer(key, axis)
1754 return self.obj._ixs(key, axis=axis)

File ~\miniconda3\envs\main\Lib\site-packages\pandas\core\indexing.py:1685, in _iLocIndexer._validate_integer(self, key, axis)
1683 len_axis = len(self.obj._get_axis(axis))
1684 if key >= len_axis or key < -len_axis:
-> 1685 raise IndexError("single positional indexer is out-of-bounds")

IndexError: single positional indexer is out-of-bounds


The script seems to run fine, and then at some random iteration df3 is empty, resulting in the error, even though df[df["PLU"]==iPLU]["ArticleName"] is not empty. I do not understand why df3 can be empty in this case.

Continue reading...