I'm training my AI model with a huge data set, so that it's impossible to preload all the data into memory at the beginning. I'm currently using psycopg2 to load data from a Postgresql DB during training. I need to convert the sample data into numpy.ndarry, while psycopg2 returns data in a tuple of tuples, where every inner tuples carry a row of data, and the outter tuple carries all. This piece of codes(loading data) is the hotest spot of entire training processing, so I want to make it as fast as possible. My current codes like this: rows = cursor.fetchall() # rows is a tuple of tuples features = [np.array( r, dtype=np.float32 ) for r in rows] features = np.stack( features, axis=0 ) # final shape of output is (128row, 78column) I'm wondering wether or not there is a faster way to convert a tuple of tuples into a 2D numpy array? Thanks! Continue reading...
