One of the main problems with callbacks is that they require you to break the program execution into small functions that will be invoked when a certain event takes place. As we saw in the earlier sections, callbacks can quickly become cumbersome.

Coroutines are another, perhaps a more natural, way to break up the program execution into chunks. They allow the programmer to write code that resembles synchronous code but will execute asynchronously. You may think of a coroutine as a function that can be stopped and resumed. A basic example of coroutines is generators.

Generators can be defined in Python using the yield statement inside a function. In the following example, we implement the range_generator function, which produces and returns values from 0 to n. We also add a print statement to log the internal state of the generator:

    def range_generator(n):
        i = 0
        while i < n:
            print("Generating value {}".format(i))
            yield i
            i += 1

When we call the range_generator function, the code is not executed immediately. Note that nothing is printed to output when the following snippet is executed. Instead, a generator object is returned:

    generator = range_generator(3)
    generator
    # Result:
    # <generator object range_generator at 0x7f03e418ba40>

In order to start pulling values from a generator, it is necessary to use the next function:

    next(generator)
    # Output:
    # Generating value 0

    next(generator)
    # Output:
    # Generating value 1

Note that every time we invoke next, the code runs until it encounters the next yield statement and it is necessary to issue another next statement to resume the generator execution. You can think of a yield statement as a breakpoint where we can stop and resume execution (while also maintaining the internal state of the generator). This ability of stopping and resuming execution can be leveraged by the event loop to allow for concurrency. 

It is also possible to inject (rather than extract) values in the generator through the yield statement. In the following example, we declare a function parrot that will repeat each message that we send. To allow a generator to receive a value, you can assign yield to a variable (in our case, it is message = yield). To insert values in the generator, we can use the send method. In the Python world, a generator that can also receive values is called a generator-based coroutine:

    def parrot():
        while True:
            message = yield
            print("Parrot says: {}".format(message))

    generator = parrot()
    generator.send(None)
    generator.send("Hello")
    generator.send("World")

Note that we also need to issue a generator.send(None) before we can start sending messages; this is done to bootstrap the function execution and bring us to the first yield statement. Also, note that there is an infinite loop inside parrot; if we implement this without using generators, we will get stuck running the loop forever!

With this in mind, you can imagine how an event loop can partially progress several of these generators without blocking the execution of the whole program. You can also imagine how a generator can be advanced only when some resource is ready, therefore eliminating the need for a callback.

It is possible to implement coroutines in asyncio using the yield statement. However, Python supports the definition of powerful coroutines using a more intuitive syntax since version 3.5.

To define a coroutine with asyncio, you can use the async def statement:

    async def hello():
        print("Hello, async!")

    coro = hello()
    coro
    # Output:
    # <coroutine object hello at 0x7f314846bd58>

As you can see, if we call the hello function, the function body is not executed immediately, but a coroutine object is returned. The asyncio coroutines do not support next, but they can be easily run in the asyncio event loop using the run_until_complete method:

    loop = asyncio.get_event_loop()
    loop.run_until_complete(coro)

The asyncio  module provides resources (called awaitables) that can be requested inside coroutines through the await syntax. For example, if we want to wait for a certain time and then execute a statement, we can use the asyncio.sleep function:

    async def wait_and_print(msg):
        await asyncio.sleep(1)
        print("Message: ", msg)
    
    loop.run_until_complete(wait_and_print("Hello"))

The result is beautiful, clean code. We are writing perfectly functional asynchronous code without all the ugliness of callbacks!

Even better, coroutines are also awaitable, and we can use the await statement to chain coroutines asynchronously. In the following example, we rewrite the network_request function, which we defined earlier, by replacing the call to time.sleep with asyncio.sleep:

    async def network_request(number):
         await asyncio.sleep(1.0)
         return {"success": True, "result": number ** 2}

We can follow up by reimplementing fetch_square. As you can see, we can await network_request directly without needing additional futures or callbacks.

    async def fetch_square(number):
         response = await network_request(number)
         if response["success"]:
             print("Result is: {}".format(response["result"]))

The coroutines can be executed individually using loop.run_until_complete:

    loop.run_until_complete(fetch_square(2))
    loop.run_until_complete(fetch_square(3))
    loop.run_until_complete(fetch_square(4))

Running tasks using run_until_complete is fine for testing and debugging. However, our program will be started with loop.run_forever most of the times, and we will need to submit our tasks while the loop is already running.

asyncio provides the ensure_future function, which schedules coroutines (as well as futures) for execution. ensure_future can be used by simply passing the coroutine we want to schedule. The following code will schedule multiple calls to fetch_square that will be executed concurrently:

    asyncio.ensure_future(fetch_square(2))
    asyncio.ensure_future(fetch_square(3))
    asyncio.ensure_future(fetch_square(4))

    loop.run_forever()
    # Hit Ctrl-C to stop the loop

As a bonus, when passing a coroutine, the asyncio.ensure_future function will return a Task instance (which is a subclass of Future) so that we can take advantage of the await syntax without having to give up the resource tracking capabilities of regular futures.