[Python] Data loading for neural network memory optimization

def load_training_data(pgn_file = 'lichess_elite_2022-02.pgn', max_games = 140000):
data = []
with open(pgn_file) as a:
for i in range(max_games):
game = chess.pgn.read_game(a)
board = game.board()
for move in game.mainline_moves():
input_tensor = board_to_tensor(board)
label = move_to_index(move)
data.append((input_tensor, label))
board.push(move)
if i % 20000 == 0 and i > 0:
print(f"Załadowano dane z {i} gier.")
print(f'Załadowano dane z {max_games} gier.')
print('Ładowanie danych ukończone.')

X = [item[0] for item in data]
y = [item[1] for item in data]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=False)

return (X_train, y_train), (X_test, y_test)


This is a function used to load data for my neural network to train and validate, I have 16GB of ram. I can load up to around 150000 games before I run out of RAM and my computer crashes. board_to_tensor converts chess library board into tensor of floats, 777 digits representing current board state and move_to_index simply gives unique numerical value for any move played. Is there any way I can optimize it RAM memory wise, cause I believe I need a lot more data for a single epoch.

Continue reading...