[Python] when to commit data in ZODB

I am trying to handel the data generated by the following piece of code:

for Gnodes in G.nodes() # Gnodes iterates over 10000 values
Gvalue = someoperation(Gnodes)
for Hnodes in H.nodes() # Hnodes iterates over 10000 values
Hvalue =someoperation(Hnodes)
score = SomeOperation on (Gvalue,Hvalue)
dic_score.setdefault(Gnodes,[]).append([Hnodes, score, -1 ])


Since the dictionary is large (10000 keys X 10000 list with 3 elements each), it is difficult to keep it in memory. I was looking for a solution which stores the key:value (in the form of list) pair as soon as they are generated. It was advised here, Writing and reading a dictionary in specific format (Python), to use ZODB in combination with Btree.

Bear with me if this is too naive, my question is, when should one call transaction.commit() to commit the data ? If I call it at the end of the inner loop, the resulting file is extremely large ( not sure why). Here is a snippet:

storage = FileStorage('Data.fs')
db = DB(store)
connection = db.open()
root = connection.root()
btree_container = IOBTree
root[0] = btree_container
for nodes in G.nodes()
btree_container[nodes] = PersistentList () ## I was loosing data prior to doing this

for Gnodes in G.nodes() # Gnodes iterates over 10000 values
Gvalue = someoperation(Gnodes)
for Hnodes in H.nodes() # Hnodes iterates over 10000 values
Hvalue =someoperation(Hnodes)
score = SomeOperation on (Gvalue,Hvalue)
btree_container.setdefault(Gnodes,[]).append([Hnodes, score, -1 ])
transaction.commit()


What if I call it outside both the loops? Something like:

......
......
score = SomeOperation on (Gvalue,Hvalue)
btree_container.setdefault(Gnodes,[]).append([Hnodes, score, -1 ])
transaction.commit()


Will all the data be held in the memory till I call transaction.commit()? Again, I am not sure why but this results in a smaller file size on disk.

I want to minimize the data being held in the memory. Any guidance would be appreciated !

Continue reading...