Notes on Generator 2

Generator

Function with a yield statement is a generation function. These generation functions are typically used to feed iteration. This looks like generator are spitting out the values, and iteration just consumes them off.

def countdown(n):
  while n > 0:
    # here it spits out n
    yield n 
    n -= 1

# consume generator in iterator
for x in countdown(10):
  # iteration consumes them
  print(x)

# consume generator step by step
In [2]: c = countdown(3)
In [3]: c
Out[3]: <generator object countdown at 0x1078f17d0>
In [4]: next(c)
Out[4]: 3
In [5]: next(c)
Out[5]: 2
In [6]: next(c)
Out[6]: 1
In [7]: next(c)
# once the countdown returns, StopIteration will be raised
# Exception StopIteration
In [8]:

This is most people know about generator, including myself. But actually, there is much more we can know about them.

As the example I wrote in the last article, the one that counts the total size of data flow from Nginx log, actually uses generators as pipelines. The idea here is we can have a stack of yield function, recursively set, and put an for-loop at the end of pipelines. The pipeline is ideal from big data processing, it keeps our memory from exploding up.

Missing Part of Generators

A yield can receive a value, see codes below.

# defines a generator function
In [1]: def receiver():
   ...:     while True:
        		# yield a value in item
   ...:         item = yield
   ...:         print item
   ...:
In [2]: r = receiver()
# first next() call to r is doing initialization, runs the function till the yield statement
# also called: advancing a generator
In [3]: next(r)
# a second next() call return nothing, cause no value was yielded
In [4]: next(r)
None
# now the yield statement received a value, so it can continue running, and prints out the item value
In [5]: r.send(1)
1
# prints out every thing it receives
In [6]: r.send([1,2,3])
[1, 2, 3]

Generators here creates coroutines, they works like actors, waiting in the background, receives values and then process them.

Besides, generator can be also be closed by calling close() on it, this method raises Generator Exit at the yield. Or we can throw exceptions by calling throw() on it.

yield from

Yield from allows a generator to call other generators.

>>> def chain(x, y):
...     yield from x
...     yield from y
>>> a = [1, 2, 3]
>>> b = [4, 5, 6]
>>> chain(a, b)
<generator object chain at 0x101c01240>
>>> for x in chain(a,b):
...     print(x)
1
2
3
4
5
6
>>> def chain2(x, y):
...     yield x
...     yield y
>>> for x in chain(a,b):
...     print(x)
[1, 2, 3]
[4, 5, 6]
>>> def chain3(x, y):
...     for i in x:
...         yield i
...     for j in y:
...         yield j
...
>>> for x in chain3(a,b):
...     print(x)
...
1
2
3
4
5
6
>>>

Reference

• Generators: The Final Frontier