I have a program to perform denormalization on large data files before I import the data to an OLAP database. I am trying to make this denormalization as fast as possible. My "native" language is Javascript so I'm doing the prototype there. The inner loop of this process, and source of 95% of the latency, is a simple string concatenation that produces each row of a csv, which I stream to a file. My question is, would another language be expected to perform this simple task much faster than javascript? The latency seems entirely CPU related. I am using parallel processes. 20 billion rows still takes about half an hour. The following code is the inner loop. I have tried very hard to tweak this in many ways to make it faster and I'm pretty sure there's no more juice to squeeze in js. Naked for loop isn't faster. Nested arrays, nope. items.forEach((id, i)=>{ rows.push( csi + "_" + id, DELIMITER, cells[0], DELIMITER, cellValues, DELIMITER, "\n" ); }); Continue reading...
