List Comprehensions and Generator Expressions

Normally, you would write a simple loop like this:

output = []
for i in range(100):
    output.append(i*i)

Whie it is good to write out complex loops for clarity, smaller loops can use list comprehensions to speed up code execution:

output = [i*i for i in range(100)]

The main speedup comes from the list append operation. There is special append bytecode just for list comprehensions.

You can also do some filtering in the list comprehension:

output = [i*i for i in range(100) if i%2 == 0]

We can also do multi-level list comprehensions:

output = [i*j for i in range(100) for j in range(100)]

In python 2.7+, sets and dicts can also be comprehensions:

# give me a set
output = {i%5 for i in range(10)}

# how about a dict?
output = {i:"test"+str(i) for i in range(10)}

Given this code:

for i in [x*x for x in range(10)]:
    # do something with i

Someone asked how to make this faster, especially if the input was very large or infinite. A generator is like a list comprehension, but without forming the list before starting to pass values to the output:

for i in (x*x for x in range(10))
    # do something with i

Note that each i is processed before the next one is even generated. This enables a streaming architecture, processing elements as they are received, and was applied to a lot of things.

For instance, big files:

for line in open('filename'):
    # process each line before loading the next one

This allows us to operate on huge files that would never fit in memory, without bothering with fancy read functions.

In Python 3, virtually anything that used to have an iterator or output a list is now a generator.

The easiest way to make a generator is by using the yield statement:

def generated_range(maximum):
    i = 0
    while i < maximum:
        yield i
        i += 1

Then we can use the generator with:

for i in generated_range(10):
    print(i)

We can also manually make a generator from a class:

class manual_range:
    def __init__(self,maximum):
        self.maximum = maximum
        self.cur = 0
    def __iter__(self):
        return self
    def __next__(self):
        return self.next()
    def next(self):
        if self.cur < self.maximum:
            ret = self.cur
            self.cur += 1
            return ret
        else:
            raise StopIteration()