[Python] What is the most efficient way to save the state of progress to non-volatile memory...

Consider a task that involves looping over an array of a million items, where each item is a string. For each iteration, the program uses the item corresponding to iteration count in that array to perform a task. For instance, consider an array containing a million customer IDs, stored as strings. The task is to loop over all the customer IDs, and for each customer ID, fetch information about the customer associated with the ID from an API and store it into a database. There is a possibility that the iteration over the loop may be interrupted due to any reason. For each iteration, upon successfully fetching the customer data from the API and storing it into the database, the program must log the state of progress in an external file. (There is no way to determine the state of progress by using the items stored in the database, or by using any database function.)

One straightforward way to do this is to store the iteration count — a single non-negative integer — to a text file and update it at the end of each iteration. Is this the most efficient way to accomplish this task? If not, what is the most efficient way to do it? Are there any libraries that provide tools for this?

Continue reading...