[Python] What is the fastest way to get distinct rows in pyarrow table?

I am trying to get information about what are the distinct combinations of values in two of the columns in my pyarrow table.

What I am currently doing is:

import pandas as pd
import pyarrow as pa
my_table = pa.Table.from_pandas(
pd.DataFrame(
{
'col1':['a', 'a', 'a', 'a', 'b', 'b', 'b', 'b'],
'col2':[1,1,2,2,1,1,2,3],
'col3':[1,2,3,4,5,6,7,8]
}
)
)
a = [i.to_numpy().astype('str') for i in my_table.select(['col1', 'col2']).columns]
unique = np.unique(np.array(a), axis = 1)


Which returns the expected result of:

unique
>array([['a', 'a', 'b', 'b', 'b'],
['1', '2', '1', '2', '3']], dtype='<U21')


but this is quite slow for larger tables and I would hope that there is a faster way?

Alternatively, what I'd really like to know, is when I am trying to write a partitioned dataset, how to know in advance to which directories will it write (i.e. which partitions have some data in my table)

EDIT:

it can be faster converting to pandas instead of multiple numpy arrays and then using drop_duplicates():

my_table.select(['col1', 'col2']).to_pandas().drop_duplicates()

Continue reading...