[Python] Why does reading large amounts of data from SQLite database using SQLAlchemy take...

I want to read millions of entries from a locally hosted database into memory in Python. Im using SQLAlchemy with SQLite. I'm indexing using a datetime string and a symbol and all queries only use these indexes. I'm happy to sacrifice write speed for read speed. I'd like the read time to be at least somewhat comparable to a csv.

Using a query and then fetchall it took me 10 mins for 10 million lines with 10 columns. Using pd.readsql it took one minute. Reading same data from csv took 18 seconds.

Continue reading...