[Python] How can I load an HDF5 file, including columns of strings, into R?

I have a bunch of pandas dataframes I'd like to save in one file, and each dataset gets appended to multiple times. I decided that an HD5 file would be a good way to store them, and this is code example of what I'm doing:

import pandas as pd
import numpy as np

letter_list = ['a', 'b', 'c', 'd']
word_list = ['apple', 'banana', 'cherry', 'date']

for i in range(3):
df1 = pd.DataFrame({'integer': np.random.randint(1, 10, 2),
'float': np.random.rand(2),
'string': np.random.choice(word_list, 2),
'letter': np.random.choice(letter_list, 2),
'run': *2})
df2 = pd.DataFrame({'integer': np.random.randint(1, 10, 2),
'float': np.random.rand(2),
'string': np.random.choice(word_list, 2),
'letter': np.random.choice(letter_list, 2),
'run': *2})

store = pd.HDFStore('test.h5')
store.append('df1', df1)
store.append('df2', df2)
store.close()


Doing this I can successfully get an hd5 file that I can read back, and get correct data from:

>>> df1 = pd.read_hdf('test.h5', key = "df1")
>>> df1
integer float string letter run
0 2 0.836773 cherry a 0
1 5 0.057874 banana b 0
0 5 0.545035 banana a 1
1 3 0.271138 banana a 1
0 6 0.086281 banana b 2
1 5 0.244409 cherry d 2


Now, I want to also be able to read this information in R as a dataframe. How can I get this full dataframe into R?

I tried using the rhdf5 as follows

library("rhdf5")
data <- h5read("../test.h5", "/df1", compoundAsDataFrame=FALSE)


but I get the following warning.

Warning: h5read for type ARRAY [STRING] not yet implemented. Values replaced by NA's.
​

It seemed like the file was read, but instead of my columns where I expect strings, there's just a column of NA's

> str(data$table)
List of 5
$ index : int [1:6(1d)] 0 1 0 1 0 1
$ values_block_0: int [1, 1:6] 2 5 5 3 6 5
$ values_block_1: num [1, 1:6] 0.8368 0.0579 0.545 0.2711 0.0863 ...
$ values_block_2: num [1:6(1d)] NA NA NA NA NA NA
$ values_block_3: int [1, 1:6] 0 0 1 1 2 2


I've also tried using the hdf5r package, as follows

library("hdf5r")
file.h5 = H5File$new("../test.h5", mode = "r")
df1 <- file.h5[["df1/table"]][]
file.h5$close_all()
str(df1)


But I don't get all my data back, the "letter" column and the "float" column have disappeared.

'data.frame': 6 obs. of 5 variables:
$ index : int 0 1 0 1 0 1
$ values_block_0: int 2 5 5 3 6 5
$ values_block_1: num 0.8368 0.0579 0.545 0.2711 0.0863 ...
$ values_block_2: chr "cherry" "banana" "banana" "banana" ...
$ values_block_3: int 0 0 1 1 2 2


Any ideas on why this is happening, or what I can do to resolve it? Or, alternatively, if there's a better way to save this data I'm all ears.

Continue reading...