Python Iterators and Generators: iter(), next() and yield

Python with AI Tutorial · Chapter 34 of 48

Python iterators and generators are the machinery behind every for loop. An iterator is an object that hands out one value at a time through next(). A generator is the easiest way to build one: a function that uses yield instead of return, producing values lazily so you can process millions of rows without loading them all into memory.

iter() and next()

Any list, tuple, string, dictionary or file can be turned into an iterator with iter(). Each call to next() returns the following value.

regions = ["North", "South", "East"]
it = iter(regions)
print(next(it))
print(next(it))
print(next(it))
print(next(it, "no more regions"))

Output:
North
South
East
no more regions

The second argument to next() is a default returned when the iterator is empty. Without it, Python raises StopIteration.

What a for loop really does

A for loop calls iter() once, then calls next() repeatedly until it catches StopIteration. You can write the same thing by hand to see the protocol in action.

sales = [3996, 2990, 1998]
it = iter(sales)
while True:
    try:
        revenue = next(it)
    except StopIteration:
        break
    print(f"Sale booked: {revenue}")

Output:
Sale booked: 3996
Sale booked: 2990
Sale booked: 1998

You will never write this loop in real code, but knowing it explains why an iterator can only be consumed once and why next() on a list fails: a list is iterable, not an iterator.

Building your own iterator class

A class becomes an iterator when it defines __iter__(), which returns the object itself, and __next__(), which returns the next value or raises StopIteration.

class InvoiceNumbers:
    def __init__(self, start, count):
        self.next_no = start
        self.remaining = count

    def __iter__(self):
        return self

    def __next__(self):
        if self.remaining == 0:
            raise StopIteration
        self.remaining -= 1
        number = f"INV-{self.next_no}"
        self.next_no += 1
        return number

for inv in InvoiceNumbers(1001, 3):
    print(inv)

Output:
INV-1001
INV-1002
INV-1003

That is twelve lines of bookkeeping for a simple counter. Generators do the same job in three.

Generator functions with yield

A function that contains yield is a generator function. Calling it does not run the body; it returns a generator object that runs up to each yield only when next() asks for a value.

def invoice_numbers(start, count):
    for i in range(count):
        yield f"INV-{start + i}"

gen = invoice_numbers(1001, 3)
print(type(gen).__name__)
print(next(gen))
print(list(gen))

Output:
generator
INV-1001
['INV-1002', 'INV-1003']

Notice that list(gen) only received the last two numbers because the first one had already been consumed by next(). Generators remember where they stopped and carry on from that point.

Here is a Python iterator class:

class InvoiceNumbers:
    def __init__(self, start, count):
        self.next_no = start
        self.remaining = count
    def __iter__(self):
        return self
    def __next__(self):
        if self.remaining == 0:
            raise StopIteration
        self.remaining -= 1
        number = f"INV-{self.next_no}"
        self.next_no += 1
        return number

Rewrite it as a generator function with yield that produces the same values, then explain in plain English which pieces of state the generator tracks automatically so I no longer need self.remaining and self.next_no.

Generators for running calculations

Because a generator keeps local variables alive between yields, it is perfect for running totals, moving averages and other calculations that depend on earlier rows.

sales = [
    {"Region": "North", "Product": "Dashboard", "Units": 4,  "Revenue": 3996, "Date": "2026-01-05"},
    {"Region": "South", "Product": "Tracker",   "Units": 10, "Revenue": 2990, "Date": "2026-01-06"},
    {"Region": "East",  "Product": "Dashboard", "Units": 2,  "Revenue": 1998, "Date": "2026-01-07"},
    {"Region": "North", "Product": "Calendar",  "Units": 15, "Revenue": 1485, "Date": "2026-01-08"},
]

def running_total(rows):
    total = 0
    for row in rows:
        total += row["Revenue"]
        yield row["Date"], row["Region"], total

for date, region, total in running_total(sales):
    print(f"{date} {region:<6} {total:>6}")

Output:
2026-01-05 North    3996
2026-01-06 South    6986
2026-01-07 East     8984
2026-01-08 North   10469

Generator expressions

A generator expression looks like a list comprehension with round brackets. It produces values on demand instead of building the whole list first, which saves memory when you only need to feed the values into sum(), max() or another consumer.

dashboard_revenue = (row["Revenue"] for row in sales if row["Product"] == "Dashboard")
print(type(dashboard_revenue).__name__)
print(sum(dashboard_revenue))
print(sum(dashboard_revenue))   # already exhausted

Output:
generator
5994
0

Processing a large file lazily

Files are iterators too: for line in f reads one line at a time. Wrapping that in a generator gives you a filter that works on a 10 GB export as comfortably as on four rows.

from pathlib import Path

Path("sales.csv").write_text(
    "Region,Product,Units,Revenue,Date\n"
    "North,Dashboard,4,3996,2026-01-05\n"
    "South,Tracker,10,2990,2026-01-06\n"
    "East,Dashboard,2,1998,2026-01-07\n"
    "North,Calendar,15,1485,2026-01-08\n", encoding="utf-8")

def large_orders(path, min_revenue):
    with open(path, encoding="utf-8") as f:
        next(f)                      # skip the header line
        for line in f:
            region, product, units, revenue, date = line.rstrip("\n").split(",")
            if int(revenue) >= min_revenue:
                yield region, product, int(revenue)

for order in large_orders("sales.csv", 2000):
    print(order)

Output:
('North', 'Dashboard', 3996)
('South', 'Tracker', 2990)

Only one line is in memory at any moment. The file is closed automatically when the generator finishes because the with block ends.

Iterator tools at a glance

The standard library ships helpers that combine well with generators. The itertools module is the one to remember.

Infinite generators with islice

Because generators are lazy, they can be infinite. Use itertools.islice() to take only what you need.

from itertools import count, islice

order_ids = count(start=5001)
print(list(islice(order_ids, 3)))
print(next(order_ids))

Output:
[5001, 5002, 5003]
5004

I have a Python script that reads a 3 GB CSV export with columns Region, Product, Units, Revenue, Date using csv.DictReader, appends every row to a list, then filters rows where Revenue > 2000 and sums Units per Region. It runs out of memory. Rewrite it using generator functions so that at most one row is in memory at a time, keep the per-Region totals in a dictionary, and explain each change in one line.

Common mistakes

• Calling next() on a list or dictionary. Only iterators support next(); wrap the object in iter() first.
• Reusing a generator after it is exhausted and wondering why the loop does nothing.
• Calling len() on a generator. Generators have no length; use sum(1 for _ in gen) or convert to a list if it is small.
• Forgetting return self in __iter__(), which makes the class unusable in a for loop.
• Looping over an infinite generator without islice() or a break, which never finishes.

Exercise

Write a generator function batches(rows, size) that yields lists of at most size items from any iterable. Test it with range(1, 8) and a batch size of 3; it should print three batches, the last one containing only 7.

def batches(rows, size):
    batch = []
    for row in rows:
        batch.append(row)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch

for b in batches(range(1, 8), 3):
    print(b)

Output:
[1, 2, 3]
[4, 5, 6]
[7]

Related chapters

• Python For Loops – the loop that drives every iterator.
• Python Classes and Objects – background for __iter__ and __next__.
• Python File Handling – reading files line by line.
• Python with AI course hub – all 48 chapters in order.

FAQ

What is the difference between an iterator and a generator in Python?

An iterator is any object with __iter__() and __next__() methods. A generator is a specific kind of iterator created automatically by a function that uses yield or by a generator expression, so you get the iterator protocol without writing the methods yourself.

What does the yield keyword do?

yield hands one value to the caller and pauses the function, keeping all local variables intact. The next call to next() resumes right after the yield line until the function ends, which raises StopIteration.

Can a generator be used more than once?

No. Once a generator has raised StopIteration it stays empty. Call the generator function again to get a fresh generator, or store the results in a list if you need to loop over them repeatedly.

Working with spreadsheets too? Ready-made Excel, Google Sheets and Power BI templates are at NextGenTemplates.com.

Chapter 34 of 48 · Python with AI: all 48 chapters