Python Iterators and Generators

Iterators:

Iterators are something that help to loop over different objects in Python. Iterator goes over all the values in the list. Iterators apply the iterator protocol that contains basically two methods:

• _iter_()

• _next_()

Before moving on to iterator protocol, let’s discuss the difference between iterators vs iterable.
Iterable objects are objects which you can iterate upon. Examples of iterables are tuples, dictionaries, lists, strings, etc.
These iterables use iter() method to fetch the iterator.

Here is an example of iter() method fetching an iterator from a tuple.

Code:

1
2
3
4
5
atuple = ('avocado', 'beetroot', 'berries')
myiter = iter(atuple)
print(next(myiter))
print(next(myiter))
print(next(myiter))

Output:
avocado
beetroot
berries

Strings are also considered iterable objects from which you can fetch an iterator.

Code:

1
2
3
4
5
astr = "beetroot"
theiter = iter(astr)
print(next(theiter))
print(next(theiter))
print(next(theiter))

Output:
b
e
e

Looping Through an Iterator:

For loop can also be used to fetch through an iterable object.

Code:

1
2
3
4
atuple = ('avocado', 'beetroot', 'berries')

for i in atuple:
  print(i)

Output:
avocado
beetroot
berries

For loop basically produces an iterator and implements the next() method for every iteration.

Iterators:

An iterator actually represents a stream of data. Iterators follow the iterator protocol.
Iterator protocol has iter() and next() methods. Both iterables and iterators have iter() method that fetches an iterator.
The thing that differentiates between iterators and iterables is the next() method, that is part of just iterators.
The next() method is used to fetch the next value of the iterable, whenever we use the print function with next.
Let’s try to use the next() method with an iterable.

Code:

1
2
3
4
subjects = ['statistics', 'machine learning', 'java', 'python']

ita = subjects.__iter__()
subjects.__next__()

Ouput:

1
2
3
4
5
6
7
8
9
<list_iterator object at 0x00000221F010CE08>
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-8f0013107d30> in <module>
      3 ita = subjects.__iter__()
      4 print(ita)
----> 5 subjects.__next__()

AttributeError: 'list' object has no attribute '__next__'

The next() method does not work with an iterable but surely works with an iterator.
Iterators can also be called iterables but not the other way around. The __next__() and __iter__() methods can be used interchangeably with next() and iter() methods.

Lets create an Iterator in Python from scratch:

Now that we have an understanding of how iterators work in Python, let’s try to get a deeper understanding from scratch.
Now we will create an iterator to print all the odd numbers.

Code:

1
2
3
4
5
6
7
8
9
class Sequence():
  def __init__(self):
  self.num = 1
def __iter__(self):
  return self
def __next__(self):
  value = self.num
self.num += 2
return value

Understanding this code:

The __init__() method is the first method that executes when any class gets called. It is used to assign values to the class variables during the program. I have declared num variable with one here.
Next and iter are the methods that make this class an iterator.
The iter method fetches the iterator and initiates the iteration. The class sequence is an iterator, thus it returns itself.
The next method fetches the current value from the iterator and moves to the next state when the next call happens. We update the num var by two to make the output odd numbers.

Lets iterate the sequence class using next method:

Code:

1
2
3
4
ite = Sequence()
print(next(ite))
print(next(ite))
print(next(ite))

Output:
1
3
5

Here I did not mention when the sequence should stop iteration. Let’s do that too.

Code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class Sequence():
  def __init__(self):
  self.num = 1
def __iter__(self):
  return self
def __next__(self):
  value = self.num
if value > 10:
  raise StopIteration

self.num += 2
return value

ite = Sequence()
print(next(ite))
print(next(ite))
print(next(ite))
print(next(ite))

Output:
1
3
5
7

Python Generators:

Generators provide a better way to create iterators in Python. This can be done by defining a proper function instead of using a return statement that uses a yield keyword. Let’s see this with the help of an example.

Code:

1
2
3
4
5
6
7
8
def subjects():
  yield "machine learning"
yield "business analytics"
yield "java"
yield "python"

for i in subjects():
  print(i)

Output:
machine learning
business analytics
java
python

The yield function is similar to a return statement, but with some additional functionality – it actually remembers the state of the function.
So next time when the generator is called it won’t start from scratch, rather from where it was last called.

Performance of a generator:

The improvement in the performance of a generator is due to the yield function, as it leads to the on-demand creation of values. We can generate the values one by one. Unlike when return terminates the state of function, yield pauses the state and continues where it was left off.

This yield function also makes the generator memory-efficient when compared with list comprehension.

Generator expressions:

Generator expressions are better at generating sequences and are memory-efficient. They are often compared to list comprehensions, as the way the code is written for both is similar.

Code:

1
2
3
4
5
6
7
#generator expression
Gen = (x ** 2
  for x in range(4))
next(Gen)
next(Gen)
next(Gen)
next(Gen)

Output:
0
1
4
9

Generator expressions are memory and space-efficient as they do not need the whole list to be generated as in list comprehension. It can be used with large datasets.

Example:

• You can start a generator using yield and next functions.

I hope the concepts of iterators, iterables, and generators are now clear to you.