I am experiencing an odd behavior with python's built-in hash() function. I am writing hashes of strings (which come from various language/text datasets) into 900 different files, according to the first three digits of their hash values. Now I noticed that all files up to 921 tend to have very similar sizes, i.e. a similar number of hashes written to them (as is to be expected); but the size of file 922 drops to around a third of that; and all files after 922 tend to have a size of only 1/10 of the other files. This happens for inherently different datasets (e.g. https://huggingface.co/datasets/embedding-data/Amazon-QA or the captions from https://cocodataset.org/#home), so it doesn't seem to be caused by the data. Shouldn't (random) strings be cast to uniformly distributed hashes? Could this be caused by the hash() function itself, i.e. is there something odd about the built-in hash function? Continue reading...
