I am copying data from postgres to SSMS, and using sqlalchemy as the go between. I read from postgress successfully, and push into a pandas dataframe. I write most of the tables to SSMS successfully, but for a few receive the following error: MemoryError I get around this by inserting 10 rows at a time. I am confused, as using sys.getsizeof(), the df in question is not larger than any of the other successfully written tables. Here is some data on the written tables. full column is the size of the entire data, partial column is the size of the chunk written to sql, if needed . 87 successful: 10 unsuccessful tables that had to be written in chunks. We see the maximum of the 10 unsuccessful is 141kb, which is 10x smaller than the maximum of the successful batch, of 1.6mb. What could be causing this MemoryError, on such a small value? redacted code: query = f'select * from {postgres_table}' cur.execute(query) df = pd.DataFrame(cur.fetchall()) # do stuff# df.to_sql(name=ssms_table, con=ssms_conxn, if_exists='append', index=False) ssms_conxn.commit() Full traceback: Traceback (most recent call last): File "<stdin>", line 49, in <module> File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\pandas\core\generic.py", line 2878, in to_sql return sql.to_sql( ^^^^^^^^^^^ File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\pandas\io\sql.py", line 769, in to_sql return pandas_sql.to_sql( ^^^^^^^^^^^^^^^^^^ File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\pandas\io\sql.py", line 1920, in to_sql total_inserted = sql_engine.insert_records( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\pandas\io\sql.py", line 1461, in insert_records return table.insert(chunksize=chunksize, method=method) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\pandas\io\sql.py", line 1023, in insert num_inserted = exec_insert(conn, keys, chunk_iter) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\pandas\io\sql.py", line 929, in _execute_insert result = conn.execute(self.table.insert(), data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\sqlalchemy\engine\base.py", line 1412, in execute return meth( ^^^^^ File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\sqlalchemy\sql\elements.py", line 483, in _execute_on_connection return connection._execute_clauseelement( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\sqlalchemy\engine\base.py", line 1635, in _execute_clauseelement ret = self._execute_context( ^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\sqlalchemy\engine\base.py", line 1844, in _execute_context return self._exec_single_context( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\sqlalchemy\engine\base.py", line 1984, in _exec_single_context self._handle_dbapi_exception( File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\sqlalchemy\engine\base.py", line 2342, in _handle_dbapi_exception raise exc_info[1].with_traceback(exc_info[2]) File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\sqlalchemy\engine\base.py", line 1934, in _exec_single_context self.dialect.do_executemany( File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\sqlalchemy\dialects\mssql\pyodbc.py", line 716, in do_executemany super().do_executemany(cursor, statement, parameters, context=context) File "C:\Users\aaa\AppData\Roaming\Python\Python311\site-packages\sqlalchemy\engine\default.py", line 918, in do_executemany cursor.executemany(statement, parameters) MemoryError Continue reading...
