I have a 200MB sized csv file containing rows where a key term is matched against a list of strings inside the second column. term_x | ["term_1","term_2"] term_y | ["term_1","term_2"] term_z | ["term_1","term_2"] My Django app is not configured to use any complex memory caching (Redis, Memcached) and in practice, I want to pass a term into the database table to retrieve the corresponding list value. Due to its size however, retrieving the list from the correct row takes around half a second to do, on top of other actions being performed while loading the page. Is it possible in Django to "pre-cache" this table upon server startup? i.e. add all of those values to the cache with the first column being the key? I have attempted something similar by overriding the "ready" method in my app.py to load the database table into the cache on startup, but I get null values when I try to use a term I know is in the table: class MyAppConfig(AppConfig): name = 'test_display' def ready(self): print("Loading RS Lookup cache..."), #setup database connection.... cache_df = pd.read_sql_query("Select * from table_to_cache", engine) print("Table loaded") for index, row in cache_df.iterrows(): cache.set(row['term'], row['list_of_terms'], None) print("RS Table loaded") My init.py in the same Django app has only one line: default_app_config = 'test_display.apps.MyAppConfig' Continue reading...
