[Python] How to use seaborn.clustermap with large (20 000 entries) data sets?

I'm working a lot with sns.clustermap and I'm quite enjoying it as an easy tool to get an overview over a large amount of data (time series data in my case, a few hundred entries mostly).

So I figured I'll enjoy it with some larger data too, but well, not so much...

I don't mind that it takes about 6 hours to calculate on my machine, but not getting any output for those 6 hours is rather annoying.



I does show the dendrogram, but the field that should contain the matrix just remains empty. Now, assuming that seaborn doesn't have inbuilt routines that deal with interpolating between pixels, this kinda makes sense. Assuming a 1 px height of a row, the matrix alone should be 20 000 pixels high with a 20 000 entries dataset. But if I increase my figsize to say figsize=(400,400) it takes a whole day and then crashes because it's out of memory and/or because my figure is too big (apparently, there's hard limit to how big a png file can be).

Is there any way to produce a clustermap with such large data sets?

When I run it with a small image size, I can produce the shown, empty picture, e.g.:

correlations = data.corr()
clusterplot = sns.clustermap(correlations, center=0
linewidths=0.4, figsize=(150, 150))
clusterplot.savefig('Cluster.png', facecolor='white', transparent=False)


With data being a pandas Dataframe with df.shape = (800, 20000).

Since it takes quite some time and doesn't produce an error, I assume clusterplot contains, well, the clusterplot, but how can I get to it? Assuming there is a way, can I also calculate the clustermap without any output? I.e. something like clusterplot = sns.clustermap(correlations, center=0, showmap=False) to save some time?

Continue reading...