[Python] How to save a generator output (dict with nested list) to the json format?

I have a python generator that yields a sample (np.ndarray, size: batchsize x ndim) of a larger dataset (np.ndarray, size: ndata x ndim).

I coded a decorator that yields a processed output (dict) of the batching generator. Something like:

def decorator(func):
def wrapper(*args, **kwargs):

generator = generator_from_large_dataset()

for kwargs['data'] in generator:
yield func(*args, **kwargs)

return wrapper
return decorator


The output of func() is a dict with nested dicts and a single nested list. Something like:

out = {'param0': 'somevalue',
'param1': {'someotheparams': 'somevalue'},
...,
'fromdata': [val1, val2, val3...]} # a list of len $batchsize


Next, I want to save $out in the json format, appending the value of $out['fromdata'] for each output of the generator. Ideally, I need to dump into the json file without reloading the last generator output because of memory issues.

I'm a newbie at generators and ijson or json_stream. What would be the best approach to save the output?

Continue reading...