[Python] Dash - Uploading extremely large .txt (3 million rows) using Upload component

New to dash, having trouble uploading my .csv files with the standard example code used here. I tried uploading normal or small size .txt files and the code works just fine. However, when I try to load very large .csv files (2 GB, close to 3 million rows and 11 columns), the page immediately breaks and shows the Google Chrome error "Aw, Snap!".

I then tried to upload the .txt in compressed .zip format (code below).


app.layout = html.Div([
dcc.Upload(
id='upload-data',
children=html.Div([
'Drag and Drop or ',
html.A('Select Files')
]),
style={
'width': '100%',
'height': '60px',
'lineHeight': '60px',
'borderWidth': '1px',
'borderStyle': 'dashed',
'borderRadius': '5px',
'textAlign': 'center',
'margin': '10px'
},
# Allow multiple files to be uploaded
multiple=True

),
html.Div(id='output-data-upload')
])

@app.callback(Output('output-data-upload', 'children'),
Input('upload-data', 'contents'),
State('upload-data', 'filename'),
State('upload-data', 'last_modified'))
def update_output(list_of_contents, list_of_names, list_of_dates):
for content, name, date in zip(list_of_contents, list_of_names, list_of_dates):
content_type, content_string = content.split(',')
content_decoded = base64.b64decode(content_string)
zip_str = io.BytesIO(content_decoded)
zip_obj = ZipFile(zip_str, 'r')

with zip_obj.open("my_original_filename.txt") as z:
df = pd.read_csv(z)
print(df.head())


The code works well in normal size compressed .csv files. However, when I pass the compressed version of my very large .csv file (which, in .zip format, now makes up 500 Mb), the list_of_contents parameter is empty. So the file is being ignored outright.

Working with very large .txt files is necessary for me, since I am working at a neuroscience lab where very long EEG readings (8 to 10 hours) are our bread and butter. How can I get this data into a Dash application?

Continue reading...