[Python] Is there any advantage to use lazy tokenizer other than space & time efficiency?

Recently I tried to implement one tokenizer following this Python doc example for Software Design for Flexibility exercise 5.7 (notice this is not one book teaching compiler specifically although it is very similar to SICP. It teaches a bit about that). That example uses iterator with lazy evaluation.

I referred to the Dragon book (I haven't read that book currently) and searched for the "lazy" context, it has no related contexts in former sections about Parsing. It has some about lazy code motion which is unrelated with lazy evaluation by iterator IMHO.

For the general case, iterator is better than list for efficiency, see QA1. This efficiency matters for big files but not for small expression string in the book exercise to be done which I am currently working with.

Both the space efficiency and the time efficiency comes from space re-usage. See the mailing list in QA1.

Iterators have a tiny constant size while lists take space proportional to the length of the list. The part that is not obvious is that looping over the iterator re-uses the same memory location again an again.


It takes time to allocate Python objects, but the smart allocator saves some of that time when it can reuse recently discarded objects.
​

Is there any advantage to use lazy tokenizer other than space & time efficiency?


p.s. Originally I asked this question about how to implement that iterator in Scheme. Later I found I haved searched about coroutine before and found this demo Scheme implementation (also see one answer posted by me related with that). The relation between coroutine and iterator is shown in PEP 492 (i.e. the former in c2 wiki means the old coroutine definition in PEP 342). So I keep scheme tag since someone may encounter with that same problem in the future.

Notice this can't be implemented by Scheme stream although that is lazy because it can't reuse space.

Continue reading...