Just going over a code snippet from this article for generator to read data from database: def generate_dataset(cursor): cursor.execute(query) for row in cursor.fetchall(): # some tramsformation yield row # 2.1. CREATE DF USING GENERATORS def create_df_gen(cursor): print('Creating pandas DF using generator...') colnames = ['transaction_id', 'user_id', 'product_name', 'transaction_date', 'amount_gbp'] df = pd.DataFrame(data = generate_dataset(cursor), columns=colnames) print('DF successfully created!\n') return df In the above code snippet we have used generator which will lazily evaluate and return one value from fetchall() call But I'm confused as to how it saves memory Considering fetchall() returns list of tuples as a result the final data is still in memory and taking up the memory which would have been the case even if we have used normal function Its basically yielding from list but the whole point of reading data from database via generator is to reduce that memory footprint since data might not be able to fit into the memory would like to understand what is the ideal way of doing it or something key I'm missing/unknown to me that makes it possible to save memory. Only approach I can think of is to read the query in paginated fashion and then use generator at that point what's the use of generator ? Continue reading...
