I need to count the number of files of multiple files .csv located in a server (250 files x 30 folders) and I would like some advice on how to do this fast, because i see its taking me a lot of time doing this for each file. After connecting with Paramiko to a sftp, my code looks like: with sftp.open('file.csv', r) as f: csv_file = csv.reader(f) sum(1 for _ in csv_file) I have also tried to get the file and then read it directly from a local folder as a dataframe (this is faster, but still slow). I have also saw there are some post in which they break the file into chunks, but I am not sure how to implement that option with a file located in a server. Continue reading...
