[Python] Pandas read_fwf error when skiprows is a function

I'm trying to read a very large fixed-width-format file and process it section-by-section.

The docs on pandas.read_fwf indicate that



However, I can't seem to make this work.

This little minimal-criminal reproduces the problem. It tries to read a similar file with four rows into two data-frames of two rows each:

import pandas as pd
try: from cStringIO import StringIO # for Python2
except ImportError: from io import StringIO # for Python3

content = '''\
Foo Bar Baz
_____ ___ ____
1.0 2 A
3.0 4 B
7.0 6 C
10.0 8 D
'''

widths = [len(s) + 1 for s in content.split('\n')[1].split()]
widths[-1] -= 1

df01 = pd.read_fwf(StringIO(content),
widths=widths,
header=0,
skiprows=[1],
nrows=2)

df23 = pd.read_fwf(StringIO(content),
widths=widths,
header=0,
skiprows=lambda r: 1 <= r <= 3,
nrows=2)


This results in the following error (trace elided):

TypeError: 'function' object is not iterable


In this case I can obviously replace the lambda by the list [1, 2, 3], but this is not practical when I'm trying to skip rows 1–1,999,999 and 3,000,000–11,000,000.

Is this a bug in pandas read_fwf? A bug in the docs?

Any suggestion for a better way to accomplish this task?

Continue reading...