10.2.1 “Closing” an iterator: the return method

Imagine a (server-side) JavaScript variant of the words() iterator above that instead of taking a source string as its argument instead takes the name of a file, opens the file, reads lines from it and iterates the words from those lines. In most operating systems, programs that open files to read from them need to remember to close those files when they are done reading, so this hypothetical iterator would be sure to close the file after the next() method returns the last word in it.

But iterators don’t always run all the way to the end: a for/of loop might be terminated with a break or return or by an exception. Similarly, when an iterator is used with destructuring assignment, the next() method is only called enough times to obtain values for each of the specified variables. The iterator may have many more values it could return, but they will never be requested.

If our hypothetical words-in-a-file iterator never runs all the way to the end, it still needs to close the file it opened. For this reason iterator objects may implement a return() method to go along with the next() method. If iteration stops before next() has returned an iteration result with the done property set to true (most commonly because you left a for/of loop early via a break statement), then the interpreter will check to see if the iterator object has a return() method. If this method exists, the interpreter will invoke it with no arguments, giving the iterator the chance to close files, release memory and otherwise clean up after itself. The return() method must return an iterator result object. The properties of the object are ignored, but it is an error to return a non-object value.

The for/of loop and the spread operator are really useful features of JavaScript, so when you are creating APIs, it is a good idea to use them when possible. But having to work with an iterable object, its iterator object and the iterator’s result objects makes the process somewhat complicated. Fortunately, generators can dramatically simplify the creation of custom iterators, as we’ll see in the rest of this chapter.

10.3.1 Generator Examples

Generators are more interesting if they actually generate the values they yield by doing some kind of computation. Here, for example, is a generator function that yields Fibonacci numbers:

function* fibonacciSequence() {
  let x = 0, y = 1;
  for(;;) {
    yield y;
    [x, y] = [y, x+y];  // Note: destructuring assignment
  }
}

Note that the fibonacciSequence() generator function above has an infinite loop and yields values forever without returning. If this generator is used with the … spread operator, it will loop until memory is exhausted and the program crashes. With care it is possible to use it in a for/of loop, however:

// Return the nth Fibonacci number
function fibonacci(n) {
  for(let f of fibonacciSequence()) {
    if (n-- <= 0) return f;
  }
}
fibonacci(20)   // => 10946

This kind of infinite generator becomes more useful with a take() generator like this:

// Yield the first n elements of the specified iterable object
function* take(n, iterable) {
  let it = iterable[Symbol.iterator](); // Get iterator for the iterable object
  while(n-- > 0) {         // Loop n times:
    let next = it.next();  // Get the next item from the iterator.
    if (next.done) return; // If there are no more values return early
    else yield next.value; // otherwise yield the value
  }
}

// An array of the first 5 Fibonacci numbers
[...take(5, fibonacciSequence())]  // => [1, 1, 2, 3, 5]

Here is another useful generator function that interleaves the elements of multiple iterable objects.

// Given an array of iterables, yield their elements in interleaved order.
function* zip(...iterables) {
  // Get an iterator for each iterable
  let iterators = iterables.map(i => i[Symbol.iterator]());
  let index = 0;
  while(iterators.length > 0) {         // While there are still some iterators
    if (index >= iterators.length)      // If we reached the last iterator
      index = 0;                        // go back to the first one.
    let item = iterators[index].next(); // Get next item from next iterator.
    if (item.done) {                    // If that iterator is done
      iterators.splice(index, 1);       // then remove it from the array.
    }
    else {                              // Otherwise,
      yield item.value;                 // yield the iterated value
      index++;                          // and move on to the next iterator.
    }
  }
}

// Interleave three iterable objects
[...zip(oneDigitPrimes(),"ab",[0])]     // => [2,'a',0,3,'b',5,7]

10.3.2 yield* and Recursive Generators

In addition to the zip() generator defined above, it might be useful to have a similar generator function that yields the elements of multiple iterable objects sequentially rather than interleaving them. We could write that generator like this:

function* sequence(...iterables) {
  for(let iterable of iterables) {
    for(let item of iterable) {
      yield item;
    }
  }
}

[...sequence("abc",oneDigitPrimes())]  // => ['a','b','c',2,3,5,7]

This process of yielding the elements of some other iterable object is common enough in generator functions that ES6 has special syntax for it. The yield* keyword is like yield except that rather than yielding a single value, it iterates an iterable object and yields each of the resulting values. The sequence() generator function above can be simplified with yield* like this:

function* sequence(...iterables) {
  for(let iterable of iterables) {
    yield* iterable;
  }
}

[...sequence("abc",oneDigitPrimes())]  // => ['a','b','c',2,3,5,7]

The array forEach() method is often an elegant way to loop over the elements of an array, so you might be tempted to write the sequence() function above like this:

function* sequence(...iterables) {
  iterables.forEach(iterable => yield* iterable );  // Error
}

This does not work, however. yield and yield* can only be used within generator functions, but nested arrow function in the code above is a regular function, not a function* generator function, so yield is not allowed.

yield* can be used with any kind of iterable object, including iterables implemented with generators. This means that yield* allows us to define recursive generators, and you might use this feature to allow simple non-recursive iteration over a recursively-defined tree structure, for example.

10.4.1 The Return Value of a Generator Function

The generator functions we’ve seen so far have not had return statements, or if they have, they have been used to cause an early return, not to return a value. Like any function, though, a generator function can return a value. In order to understand what happens in this case, recall how iteration works. The return value of the next() function is an object that has a value property and/or a done property. With typical iterators and generators, if the value property is defined, then the done property is undefined or is false. And if done is true then value is undefined. But in the case of a generator that returns a value, the final call to next returns an object that has both value and done defined. The value property holds the return value of the generator function and the done property is true indicating that there are no more values to iterate. This final value is ignored by the for/of loop and by the spread operator, but it is available to code that manually iterates with explicit calls to next():

function *oneAndDone() {
  yield 1;
  return 'done'
}

// The return value does not appear in normal iteration.
[...oneAndDone()]   // => [1]

// But it is available if you explicitly call next()
let generator = oneAndDone();
generator.next()           // => { value: 1, done: false}
generator.next()           // => { value: 'done', done: true }
// If the generator is already done, the return value is not returned again
generator.next()           // => { value: undefined, done: true }

10.4.2 The Value of a yield Expression

In the discussion above, we’ve treated yield as a statement that takes a value, but has no value of its own. In fact, however, yield is an expression and it can have a value.

When the next() method of a generator is invoked, the generator function runs until it reaches a yield expression. The expression that follows the yield keyword is evaluated and that value becomes the return value of the next() invocation. At this point, the generator function stops executing right in the middle of evaluating the yield expression. The next time the next() method of the generator is called, the argument passed to next() becomes the value of the yield expression that was paused. So the generator returns values to its caller with yield, and the caller passes values in to the generator with next(). The generator and caller are two separate streams of execution passing values (and control) back and forth. The following code illustrates:

function* smallNumbers() {
  console.log('next() invoked the first time; argument discarded');
  let y1 = yield 1;    // y1 == 'b'
  console.log('next() invoked a second time with argument', y1);
  let y2 = yield 2;    // y2 == 'c'
  console.log('next() invoked a third time with argument', y2);
  let y3 = yield 3;    // y3 == 'd'
  console.log('next() invoked a fourth time with argument', y3);
  return 4;
}

let g = smallNumbers();
console.log('generator created; no code runs yet');
let n1 = g.next('a');   // n1.value == 1
console.log('generator yielded', n1.value);
let n2 = g.next('b');   // n2.value == 2
console.log('generator yielded', n2.value);
let n3 = g.next('c');   // n3.value == 3
console.log('generator yielded', n3.value);
let n4 = g.next('d');   // n4 == { value:4, done:true }
console.log('generator returned', n4.value);

Note the asymmetry in the code above. The first invocation of next() starts the generator, but the value passed to that invocation is not accessible to the generator.

10.4.3 The return() and throw() Methods of a Generator

We’ve seen above that you can receive values yielded by or returned by a generator function. And you can pass values to a running generator by passing those values when you call the next() method of the generator.

In addition to providing input to a generator with next(), you can also alter the flow of control inside the generator by calling its return() and throw() methods. As the names suggest, calling these methods on a generator causes it to return a value or throw an exception as if the next statement in the generator was a return or throw.

let smallPrimes = oneDigitPrimes();
smallPrimes.next()     // => {value: 2, done: false}
smallPrimes.return(11) // => {value: 11, done: true}
try {
    smallPrimes = oneDigitPrimes();
    smallPrimes.throw(new Error(13));
}
catch(e) {
    // e.message == '13'
}

Recall from earlier in the chapter that if an iterator defines a return() method and iteration stops early then the interpreter automatically calls the return() method to give the iterator a chance to close files or do other cleanup. In the case of generators, you can’t define a custom return() method to handle cleanup, but you can structure the generator code to use a try/finally statement that ensures the necessary cleanup is done (in the finally block) when the generator returns. By forcing the generator to return, the generator’s built-in return() method ensures that the cleanup code is run when the generator will no longer be used.

Just as the next() method of generator allows us to pass arbitrary values into a running generator, the throw() method of a generator gives us a way to send arbitrary signals (in the form of exceptions) into a generator. Calling the throw() method always causes an exception inside the generator. But if the generator function is written with appropriate exception handling code, the exception need not be fatal but can instead be a means of altering the behavior of the generator. Imagine, for example, a counter generator which yields an ever-increasing sequence of integers. This could be written so that an exception sent with throw() would reset the counter to zero.

When a generator uses yield* to yield values from some other iterable object, then a call to the next() method of the generator causes a call to the next() method of the iterable object. The same is true of the return() and throw() methods. If a generator uses yield* on an iterable object that has these methods defined, then calling return() or throw() on the generator causes the iterator’s return() or throw() method to be called in turn. All iterators must have a next() method. Iterators that need to clean up after incomplete iteration should define a return() method. And any iterator may define a throw() method, though I don’t know of any practical reason to do so.

10.4.4 A Final Note about Generators

Generators are a very powerful generalized control structure. They give us the ability to pause a computation with yield and restart it again at some arbitrary later time with an arbitrary input value. It is possible to use generators to create a kind of cooperative threading system within single-threaded JavaScript code. And it is possible to use generators to mask asynchronous parts of your program so that your code appears sequential and synchronous even though some of your function calls are actually asynchronous and depend on events from the network.

Trying to do these things with generators leads to code that is mind-bendingly hard to understand or to explain. It has been done, however, and the only really practical use-case has been for managing asynchronous code. JavaScript now has async and await keywords for this very purpose, however (see Chapter 11), and there is no longer any reason to abuse generators in this way.