I am creating processed images for fine-tuning a custom control net. The data size is ~20K images. Since this are conditioning images they are created through a processing function the flow basically looks like this: def process(image, conditioning, filename): im1 = cond1(image, conditioning) cv2.imwrite(f"filename{1}.png", im1) im2 = cond2(image, conditioning) cv2.imwrite(f"filename{2}.png", im2) im3 = cond3(image, conditioning) cv2.imwrite(f"filename{3}.png", im3) im4 = cond4(image, conditioning) cv2.imwrite(f"filename{4}.png", im4) def main(): for image, conditioning in zip(images, conditions, filenames): process(image, conditioning, filename) I have optimised the codes that generate the images in process. But the bottleneck seems to be imwrite. Also the files are really small. On an average they are 10-20 kb 5-10% might be more than 100 kb. I've tried switching to jpeg and change compression. But that does not help much either. Is there a multiprocessing method of doing this that does not require holding the images in a queue? Current process takes about 2-5 secs per image. Continue reading...
