Here's an interesting problem: given a large amount of text data (~5 GB of words as strings), I need to build a mapping such that every word is associated with a unique integer. It should be noted that it needs to work the other way - every integer should also be associated with a unique word (hence why its a bijective mapping). I also need to be able to quickly look up a word by its associated number. The following is the most naive implementation I can think of: data_structure = [] for word in giant_list_of_words: if (word not in data_structure): data_structure.append(word) return data_structure def lookup(data_structure, i): return data_structure With this approach, the mapping is simply words to their index in the list. Building the mapping is slow, but lookup is fast. Here's another approach: def mapping(): data_structure = {} count = 0 for word in giant_list_of_words: if (word not in data_structure): data_structure[word] = count count += 1 return data_structure def lookup(data_structure, i): retval = '' for key in data_structure: if (data_structure[key] == i): retval = key break return retval This gets built fast, but is slow to index. Any thoughts? Continue reading...
