I'm trying to write a program to search an extremely large text file for specific strings, and then return the string as well as a certain amount of characters before and after it from within the file, and to be able to quickly do this thousands of times. The file is all one line and over a gigabyte in size, and most approaches I have seen work by breaking the file up into lines. I want to find a way to search efficiently AND take advantage of multiple threads. I have been looking into trying to use the Polars library but I am not sure if it will work for just a long text file. This is what I have written so far, which works, but not very quickly with open(r"file.txt", 'rb', 0) as file: for date in D: x = date.encode('UTF-8') s = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) Pos = s.find(x) if Pos != -1: file.seek(Pos-30) P.append(file.read(60 + len(date)).decode('UTF-8')) else: print(date + " was not found") file.close() Continue reading...
