The algorithm behind building a speech detector has two components:

• The generation model
• The convolution queues

These two components are shown in the following diagram:

The generation model can be viewed as one step of an RNN. It takes the current observation and several recurrent states as input, and then computes the output prediction and new recurrent states.

The convolution queues store the new recurrent states that have been computed by the layer underneath it.

Let's jump into building the model.