[Python] What is holding the GIL and how can it be released

from concurrent.futures import ThreadPoolExecutor
import time
import io
import pandas as pd
from image_exporter import export
import numpy as np

df1 = pd.DataFrame(np.random.rand (6, 4))
df2 = pd.DataFrame(np.random.rand (6, 4))
df3 = pd.DataFrame(np.random.rand (6, 4))
df4 = pd.DataFrame(np.random.rand (6, 4))

def get_exported(df):
s = time.time()
buffer = io.BytesIO()
export(df.style.background_gradient(), buffer)
buffer.seek(0)
print('get_exported', time.time() - s)

def test_parallel():
tasks = [df1, df2, df3, df4]
with ThreadPoolExecutor() as executor:
executor.map(get_exported, tasks)

def test_seq():
tasks = [df1, df2, df3, df4]
for t in tasks:
get_exported(t)

s = time.time()
test_parallel()
print('Total time parallel', time.time() - s)

s = time.time()
test_seq()
print('Total time sequential', time.time() - s)


image_exporter file that contains the export method https://gist.github.com/vukrado/bd14192984c51569f6ed94bcd49f4b24

The test_seq on my machine takes ~0.035 seconds per call to get_exported for a total run time of ~.14 seconds

The test_parallel on my machine takes ~0.18 seconds per call to get_exported for a total run time of ~.19 seconds

Continue reading...