[Python] When reading a database table with polars, how do I avoid a SchemaError?

I have a large table_to_load in a database file my_database.db that I am trying to read into a Python program as a polars DataFrame. Here is the code that does the reading:

import polars as pl

conn = sqlite3.connect('my_database.db')
df = pl.read_database(connection=conn, query='SELECT * FROM table_to_load', infer_schema_length=None)
conn.close()


When I run this code, pl.read_database throws the error, "polars.exceptions.SchemaError: failed to determine supertype of i64 and binary." The traceback is at the end of the post. I have several closely related questions about this:

1. Each column of table_to_load verifiably already has one of the three sqlite3 datatypes integer, real, and text. Can I instruct polars to just use a reasonable analog of each datatype? For example, i64 seems like a reasonable analog of integer.
2. Why does polars struggle to determine the datatype if it is reading the whole column as instructed by the parameter setting infer_schema_length=None, and is there any way to resolve this? (It looks like there is also a schema_overrides parameter, but this would be challenging to use in my application because my table columns are not static.)
3. In the traceback, why does polars not reveal the column or columns that are generating the SchemaError, and is there a way to request that it do so? In my case, this would be helpful because table_to_load has hundreds of columns.

The traceback of the SchemaError:

Traceback (most recent call last):
File "c:\Users\user0\AppData\Local\Programs\Python\Python312\test_module.py", line 115, in <module>
main_function()
File "c:\Users\user0\AppData\Local\Programs\Python\Python312\test_module.py", line 54, in main_function
df = pl.read_database(connection=conn, query='SELECT * FROM table_to_load', infer_schema_length=None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user0\AppData\Local\Programs\Python\Python312\Lib\site-packages\polars\io\database\functions.py", line 251, in read_database
).to_polars(
^^^^^^^^^^
File "C:\Users\user0\AppData\Local\Programs\Python\Python312\Lib\site-packages\polars\io\database\_executor.py", line 543, in to_polars
frame = frame_init(
^^^^^^^^^^^
File "C:\Users\user0\AppData\Local\Programs\Python\Python312\Lib\site-packages\polars\io\database\_executor.py", line 300, in _from_rows
return frames if iter_batches else next(frames) # type: ignore[arg-type]
^^^^^^^^^^^^
File "C:\Users\user0\AppData\Local\Programs\Python\Python312\Lib\site-packages\polars\io\database\_executor.py", line 283, in <genexpr>
DataFrame(
File "C:\Users\user0\AppData\Local\Programs\Python\Python312\Lib\site-packages\polars\dataframe\frame.py", line 377, in __init__
self._df = sequence_to_pydf(
^^^^^^^^^^^^^^^^^
File "C:\Users\user0\AppData\Local\Programs\Python\Python312\Lib\site-packages\polars\_utils\construction\dataframe.py", line 461, in sequence_to_pydf
return _sequence_to_pydf_dispatcher(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user0\AppData\Local\Programs\Python\Python312\Lib\functools.py", line 909, in wrapper
return dispatch(args[0].__class__)(*args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user0\AppData\Local\Programs\Python\Python312\Lib\site-packages\polars\_utils\construction\dataframe.py", line 674, in _sequence_of_tuple_to_pydf
return _sequence_of_sequence_to_pydf(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user0\AppData\Local\Programs\Python\Python312\Lib\site-packages\polars\_utils\construction\dataframe.py", line 590, in _sequence_of_sequence_to_pydf
pydf = PyDataFrame.from_rows(
^^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.SchemaError: failed to determine supertype of i64 and binary


[1]: https://www.sqlite.org/datatype3.html

Continue reading...