I have a specific question about how I can implement my following problem most efficiently. Suppose n = 100.000 and for each i,j from 1 to n I have a list containing integers between 1 and n (they don't have to be unique), i.e. essentially a matrix of lists, lets call it A. Each list is rather small, say the maximum number of items is something like 20. Essentially, for two n-dimensional arrays aand b, I want to do the following: Initialise some c and calculate for i in range(n): for j in range(n): tmp_list = A[i,j] # this corresponds to some indices for k in tmp_list: c[k] += a * b[j] I can implement this in python without any problem, for smaller n. However for such large n I have to think about multiple things: Suppose I can precompute A. How should I store it? Should I switch to another language like C? For generic a and b, is n=100.000 a dimension which is still reasonable. I.e. can I expect that I can calculate c in a reasonable amount of time, when I implement this rather optimally, or is this hopeless? I am not a very experienced programmer, have you some feeling or ideas to the above questions, in particular: Should I switch to C? How/Where should I store the data A to access it optimally during computation? Can I expect this to work at all for such dimensions? Thanks in advance! Continue reading...
