Residual Connections

Residual connections are the first step toward nonlinearity – these are simple connections placed between nonadjacent layers. They’re often presented as “skipping” over a layer or multiple layers, which is why they are also often referred to as “skip connections” (Figure 6-4).

Note that because the Keras Functional API cannot define multiple layers as the input to another layer, in implementation the connections are first merged through a method like adding or concatenation. The merged components are then passed into the next layer (Figure 6-5). This is the implicit assumption of all residual connection diagrams.

Although residual connections are incredibly versatile tools, there are generally two methods of residual connection usage, based on the ResNet and DenseNet architectures that pioneered their respective usages of residual connections. “ResNet-style” residual connections employ a series of short residual connections that are repeated routinely throughout the network (Figure 6-6).

“DenseNet-style” residual connections, on the other hand, place residual connections between valid layers (i.e., layers designated as open to having residual connections attached, also known as “anchor points” or “anchor layers”) (Figure 6-7). Because – as one might imagine – this sort of usage of residual connections leads to a high number of residual connections, seldom do DenseNet-style architectures treat all layers as residual connections. In this style, both long and short residual connections are used to provide information pathways across sections of the architecture. Because each anchor point is connected to each anchor point before it, it is informed by various stages of the network’s processing and feature extraction.

This representation of residual connections (Figure 6-4) is probably the most convenient interpretation of what a residual connection is for the purposes of implementation. It visualizes residual connections as nonlinearities added upon a “main sequence” of layers – a linear “backbone” upon which residual connections are added. In the Keras Functional API, it’s easiest to understand a residual connection as “skipping” over a “main sequence” of layers. In general, implementing nonlinearities is best performed with this concept of a linear backbone in mind, because code is written in a linear format that can be difficult to translate nonlinearities to.

However, there are other architectural interpretations of what a residual connection is that may be conceptually more aligned with the general class of nonlinearities in neural network architectures. Rather than relying upon a linear backbone, you can interpret a residual connection as splitting the layer before it into two branches, which each process the previous layer in their own unique ways. One branch (Layer 1 to Layer 2 to Layer 3 in Figure 6-8) processes the output of the previous layer with a specialized function, whereas the other branch (Layer 1 to Identity to Layer 3) processes the output of the layer with the identity function – that is, it simply allows the output of the previous layer to pass through, the “simplest” form of processing (Figure 6-8).

This method of conceptually understanding residual connections more easily allows you to categorize them as a sub-class of general nonlinear architectures, which can be understood as a series of branching structures (Figure 6-9). We’ll see how this interpretation helps later in our exploration of parallel branches and cardinality.

Residual connections are often presented as a technical justification for the vanishing gradient problem (Figure 6-10), which shares many characteristics to the previously discussed problem of a linear arrangement of thinkers: in order to access some layer, we need to travel through several other layers first, diluting the information signal. In the vanishing gradient problem, the backpropagation signal within very deep neural networks used to update the weights gets progressively weaker such that the front layers are barely utilized at all.

With residual connections, however, the backpropagation signal travels through fewer average layers to reach some particular layer’s weights for updating. This enables a stronger backpropagation signal that is better able to make use of the entire model architecture.

There are other interpretations of residual connections too. Like the random forest algorithm is constructed of many smaller decision tree models trained on part of the dataset, a neural network with a sufficient number of residual connections can be thought to be an “ensemble” of smaller sequential models built with a fewer number of layers (Figure 6-11).

Residual connections can also be thought of as a “failsafe” for poor performing layers. If we add a residual connection from layer A to layer C (assuming layer A is connected to layer B, which is connected to layer C), the network can “choose” to disregard layer B by learning near-zero weights for connections from A to B and from B to C while information is channeled directly from layer A to C via a residual connection. In practice, however, residual connections act more as an additional representation of the data for consideration than as failsafe mechanisms.

Implementing individual residual connections is quite simple with our knowledge of the Functional API. We’ll use the first presented interpretation of the residual connection architecture, in which a residual connection acts as a “skipping mechanism” between nonadjacent layers in a linear architecture backbone. For simplicity, let’s define this linear architecture backbone as a series of Dense layers (Listing 6-1, Figure 6-12).

inp = L.Input((128,))

layer1 = L.Dense(64, activation='relu')(inp)

layer2 = L.Dense(64, activation='relu')(layer1)

layer3 = L.Dense(64, activation='relu')(layer2)

output = L.Dense(1, activation='sigmoid')(layer3)

model = keras.models.Model(inputs=inp, outputs=output)

Listing 6-1

Creating a linear architecture using the Functional API to serve as the linear backbone for residual connections

Let’s say that we want to add a residual connection from layer1 to layer3. In order to do this, we need to merge layer1 with whatever current layer is the input to layer3 (this is layer2). The result of the merging is then passed as the input to layer3 (Listing 6-2, Figure 6-13).

inp = L.Input((128,))

layer1 = L.Dense(64, activation='relu')(inp)

layer2 = L.Dense(64, activation='relu')(layer1)

concat = L.Concatenate()([layer1, layer2])

layer3 = L.Dense(64, activation='relu')(concat)

output = L.Dense(1, activation='sigmoid')(layer3)

model = keras.models.Model(inputs=inp, outputs=output)

Listing 6-2

Building a residual connection by adding a merging layer

However, this method of manually creating residual connections by explicitly defining merging and input flows is inefficient and not scalable (i.e., it becomes too unorganized and difficult if you wanted to define dozens or hundreds of residual connections). Let’s create a function, make_rc() (Listing 6-3), that takes in a connection split layer (this is the layer that “splits” – there are two connections stemming from it) and a connection joining head (this is the layer before the layer in which the residual connection and the “linear main sequence” join together) and outputs a merged version of those two layers that can be used as an input to the next layer (Figure 6-14). We’ll see how automating the creation of residual connections will be incredibly helpful soon when we attempt to construct more elaborate ResNet- and DensNet-style residual connections.

We can automate the usage of this function to create a ResNet-style architecture in which blocks of layers with short residual connections are repeated several times (Figure 6-15). In order to automate the construction of the architecture, we will use the placeholder variables x, x1, and x2. We will build x1 from x and x2 from x1 in sequences and merge x with x2 to build the residual connection (Listing 6-5).

Since at the end of the building iterations x is the last connected layer, we connect x to the output layer and aggregate the architecture into a model (Listing 6-6, Figure 6-16).

# build output

output = L.Dense(1, activation='sigmoid')(x)

# aggregate into model

model = keras.models.Model(inputs=inp, outputs=output)

Listing 6-6

Building output and aggregating ResNet-style architecture into a model

A DenseNet residual connection pattern (Figure 6-17), in which every “anchor point” is connected to every other “anchor point,” requires some more planning . (Note that for the sake of ease, we will build residual connections between every Dense layer rather than building additional un-connected layers. We’ll discuss residual connections for cell-based architectures in the next section.) We’ll keep a list of previous layers x. Every building step, we add a new layer x[i] that is connected to the merging of every previous layer. Note that the layer x[i-1] is the direct connection and the connection between x[i-2], x[i-3], … and x[i] is a residual connection.

We can begin by defining an initial layer x and a list of created layers layers. After creating the initial layer, we loop through the remaining layers to be added by redefining the template variable x as a Dense layer that takes in a merged version of all the other currently created layers (Listing 6-8, Figure 6-18). Afterward, we append x to layers such that the next created layer will take in this just created layer.

# define number of Dense layers

num_layers = 5

# create input layer

inp = L.Input((128,))

x = L.Dense(64, activation='relu')(inp)

# set layers list

layers = [x]

# loop through remaining layers

for i in range(num_layers-1):

    # define new layer

    x = L.Dense(64)(make_rc(layers))

    # add layer to list of layers

    layers.append(x)

# add output

output = L.Dense(1, activation='sigmoid')(x)

# build model

model = keras.models.Model(inputs=inp, outputs=output)

Listing 6-8

Using the augmented residual-connection-making function to create DenseNet-style residual connections

This is a great example of how using predefined functions, storage, and built-in scalability in combination with one another allows for a quick and efficient building of complex topologies.

As you can see, when we begin to build more complex network designs, the Keras plotting utility starts to struggle to produce an architecture visualization that is visually consistent with our conceptual understanding of the topology. Regardless, visualization still serves as a sanity check tool to ensure that your automation of complex residual connection relationships is functioning.

Branching and Cardinality

The concept of cardinality is the core framework of nonlinearity and a generalization of residual connections. While the width of a network section refers to the number of neurons in the corresponding layer(s), the cardinality of a network architecture refers to the number of “branches” (also known as parallel towers) in a nonlinear space at a certain location in a neural network architecture.

Cardinality is most clear in parallel branches – an architectural design in which a layer is “split” into multiple layers, each of which are processed linearly and eventually merged back together. The cardinality of a segment of a network employing parallel branches is simply the number of branches (Figure 6-19).

As mentioned prior in the section on residual connections, one can generalize residual connections as a simple branching mechanism with a cardinality of two, in which one branch is the series of layers the residual connection “skips over” and the other branch is the identity function.

Note that, depending on the specific topology, the cardinality of a section of a network is more or less ambiguous. For instance, some topologies may build branches within sub-branches and join together certain sub-branches in complex ways (e.g., Figure 6-20). Here, the specific cardinality of the network is not relevant; what is relevant is that information is being modeled in a nonlinear fashion that encourages multiplicity of representation (the general concept of cardinality) and thus greater complexity and modeling power.

It should be noted, though, that nonlinearities should not be built arbitrarily. It can be easy to get away with building complex nonlinear topologies for the sake of nonlinearity, but you should have some idea of what purpose the topology serves to enable. Likewise, a key component of the network design is not just the architecture within which layers reside but which layers and parameters are chosen to fit within the nonlinear architecture. For instance, you may allocate different branches to use different kernel sizes or to perform different operations to encourage multiplicity of representation. See the case study in the next section on cell-based design for an example on good purposeful design of more complex nonlinearities, the Inception cell.

The logic of branch representations and cardinality in architectural designs is very similar to that of residual connections. This sort of representation is the natural architectural development as a generalization of the residual connection – rather than passing the information flowing through the residual connection (i.e., doing the “skipping”) through an identity function, it can be processed separate from other components of the network.

In our analogy of a network as a linked set of thinkers engaged in a dialogue whose net output is dependent on their arrangement, branch representations allow not only for thinkers to consider multiple perspectives but for entire “schools of thought” to emerge in conversation with one another (Figure 6-21). Branches process information separately (i.e., in parallel), allowing different modes of feature extraction to become “mature” (fully formed by several layers of processing) before they are merged with other branches for consideration.

Let’s begin by creating a multi-branch nonlinear topology explicitly/manually (Listing 6-9). We will create two branches that span from a layer; each branch, which holds a particular representation of the data, is processed independently and merged later (Figure 6-22).

inp = L.Input((128,))

layer1 = L.Dense(64)(inp)

layer2 = L.Dense(64)(layer1)

branch1a = L.Dense(64)(layer2)

branch1b = L.Dense(64)(branch1a)

branch1c = L.Dense(64)(branch1b)

branch2a = L.Dense(64)(layer2)

branch2b = L.Dense(64)(branch2a)

branch2c = L.Dense(64)(branch2b)

concat = L.Concatenate()([branch1c, branch2c])

layer3 = L.Dense(64, activation='relu')(concat)

output = L.Dense(1, activation='sigmoid')(layer3)

model = keras.models.Model(inputs=inp, outputs=output)

Listing 6-9

Creating parallel branches manually

Building an entire series of parallel branches within a neural network architecture is now incredibly simple (Listing 6-12, Figure 6-23).

inp = L.Input((128,))

layer1 = L.Dense(64)(inp)

layer2 = L.Dense(64)(layer1)

layer3 = L.Dense(64)(build_branches(layer2))

output = L.Dense(1)(layer3)

model = keras.models.Model(inputs=inp, outputs=output)

Listing 6-12

Building parallel branches into a complete model architecture

This sort of method is used by the ResNeXt architecture, which employs parallel branches as a generalization and “next step” from residual connections.

Case Study: U-Net

The goal of semantic segmentation is to segment, or separate, various items within an image into various classes. Semantic segmentation can be used to identify items in a picture, like cars, people, and buildings in a picture of the city. The difference between semantic segmentation and a task like image recognition is that semantic segmentation is an image-to-image task, whereas image recognition is an image-to-vector task. Image recognition tells you if an object is present in the image or not; semantic segmentation tells you where the object is located in the image by marking each pixel as part of the corresponding class or not. The output of semantic segmentation is called the segmentation map .

Semantic segmentation has many applications in biology, which can be used to automate the identification of cells, organs, neuron links, and other biological entities (Figure 6-24). As such, much of research into semantic segmentation architectures has been developed with these biological applications in mind.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox proposed the U-Net architecture in 2015,¹ which has since become a pillar of semantic segmentation development (Figure 6-25). The linear backbone component of the U-Net architecture acts like an autoencoder – it successively reduces the dimension of the image until it reaches some minimum representation size, upon which the dimension of the image is successively increased via upsampling and up-convolutions. The U-Net architecture employs very large residual connections that connect large parts of the network together, connecting the first block of layers to the last block, the second block to the second-to-last block, etc. When the residual connections are arranged to be parallel to one another in an architectural diagram, the linear backbone is forced into a “U” shape, hence its name.

The left side of the architecture is termed the contracting path ; it develops successively smaller representations of the data. On the other hand, the right side is termed the expansive path , which successively increases the representation of the data. Residual connections allow the network to incorporate previously processed representations into the expansion of the representation size. This architecture allows both for localization (focus on a local region of the data input) and the use of broader context (information from farther, nonlocal regions of the image that still provide useful data).

While the U-Net architecture is technically adaptable to all image sizes because it is built using only convolution-type layers that do not require a fixed input, the original implementation by Ronneberger et al. requires significant planning with respect to the shape of information as it passes through the network. Because the proposed U-Net architecture does not use padding, the spatial resolution is successively reduced and increased divisively (by max pooling) and additively (by convolutions). This means that residual connections must include a cropping function in order to properly merge earlier representations with later representations that have a smaller spatial size. Moreover, the input shape must be carefully planned to have a certain input size, which will not match the output.

Thus, in our implementation of the U-Net architecture, we will slightly adapt the convolutional layers to use padding such that merging and keeping track of data shapes is simpler. While implementing U-Net is relatively simple, it’s important to keep track of variables. We will name our layers with appropriate variable names such that they can be used in the make_rc() function earlier discussed to be part of a residual connection.

As mentioned earlier, you can change the input shape and the code will produce a valid result, as long as the spatial dimensions of the input are divisible by 2. You can plot the model with plot_model to reveal the architectural namesake of the model (Figure 6-26).

Sequential Cell Design

Sequential cell designs are – as implied by the name – cell architectures that follow a sequential, or linear, topology. While sequential cell designs have not generally performed as well as nonlinear cell designs, they can be useful as a beginning location to illustrate key concepts that will be used in nonlinear cell designs.

These cells can then be repeatedly stacked by iteratively passing the output of the previously constructed cell as the input to the next cell (Listing 6-18). This combination of cell design and stacking – linear cell design stacked in a linear fashion – is perhaps the simplest cell-based architectural design (Figure 6-29).

num_cells = 3

inp = L.Input((128,))

x = build_static_dense_cell(inp)

for i in range(num_cells-1):

    x = build_static_dense_cell(x)

output = L.Dense(1, activation='sigmoid')(x)

model = keras.models.Model(inputs=inp, outputs=output)

Listing 6-18

Stacking static dense cells together

Note that while we are building a cell-based architecture, we’re not using compartmentalized design (introduced in Chapter 3 on autoencoders). That is, while conceptually we understand that the model is built in terms of repeated block segments, we do not build that into the actual implementation of the model by defining each cell as a separate model. In this case, there is no need for compartmentalized design since we do not need to access the output of any one particular cell. Moreover, defining builder functions should serve as a sufficient code-level conceptual organization of how the network is constructed. The primary burden of using compartmentalized design with cell-based structures is the need to keep track of the input shape of the previous cell’s output in an automated fashion, which is required when defining separate models. However, implementing compartmentalized design will make Keras’ architecture visualizations more consistent with our conceptual understanding of cell-based models by visualizing cells rather than the entire sequence of layers.

Note that static dense and static convolutional cells have different effects on the data input shapes they are applied to. Static dense cells always output the same data output shape, since the user specifies the number of nodes the input will be projected to when defining a Dense layer in Keras. On the other hand, static convolutional cells output different data output shapes, depending on which data input shapes they received. (This is discussed in more detail in Chapter 2 on transfer learning.) Because different primary layer types in cell designs can yield different impacts on the data shape, it is general convention not to build static cells.

Instead, cells are generally built by their effect on the output shape, such that they can more easily be stacked together. This is especially helpful in nonlinear stacking patterns, in which cell outputs must match to be merged in a valid fashion. Generally, cells can be categorized as reduction cells or normal cells. Normal cells keep the output shape the same shape as the input shape, whereas reduction cells decrease the output shape from the input shape. Many modern architectures employ multiple designs for normal and reduction cells that are stacked repeatedly throughout the model.

We can simply sequentially stack these two cells together in an alternating pattern (Listing 6-22, Figure 6-31). We use the holder variables cell_out and w to keep track of the output of a cell and the corresponding width. Each cell-building function either keeps the shape w the same or modifies it to reflect changes in the shape of the output layer of the cell. This information is, as mentioned before, passed into the following cell-building functions.

num_repeats = 2

w = 128

inp = L.Input((w,))

cell_out, w = build_normal_cell(inp, w)

for repeat in range(num_repeats):

    cell_out, w = build_reduce_cell(cell_out, w)

    cell_out, w = build_normal_cell(cell_out, w)

output = L.Dense(1, activation='sigmoid')(cell_out)

model = keras.models.Model(inputs=inp, outputs=output)

Listing 6-22

Stacking reduction and normal cells linearly

Building convolutional normal and reduction cells is similar, but we need to keep track of three shape elements rather than one. Like in the context of autoencoders and other shape-sensitive contexts, it’s best to use padding='same' to keep the input and output shapes the same.

These two cells can then be stacked in a linear arrangement, as previously demonstrated. Alternatively, these cells can be stacked in a nonlinear format. To do this, we’ll use the methods and ideas developed from the nonlinear and parallel representation section.

To merge the outputs of two cells together, they need to have the same shape. Thanks to shape-based design, we can arrange normal and reduction cells accordingly to form valid merge operations. For instance, we can draw a residual connection from a normal cell “over” another normal cell, merge the two connections, and pass the merged result into a reduction cell as displayed in Figure 6-32.

However, reduction cells form “borders” that cannot be transgressed by connections (unless you use reshaping mechanisms), because the input shapes on the two sides of the reduction cell do not match (Figure 6-33).

We can build nonlinear cell-stacking architectures in a very similar way as we built nonlinearities and parallel representations among layers in a network architecture. We will use the make_rc function defined in the previous section to merge the outputs of norm_1 and norm_2 and pass the merged result as the input of a reduction cell (Listing 6-25, Figure 6-34). The shape is defined after the input with a depth of 64 such that convolutional layers process the data with 64 filters (rather than 3). If desired, you can manipulate reduction cells to also change the image depth. Note that the merging operation we use in this case is adding rather than concatenation. The output shape of adding is the same as the input shape of any one of the inputs, whereas depth-wise concatenation changes the shape. This can be accommodated but requires ensuring the shape is correspondingly updated.

inp = L.Input((128,128,3))

shape = (128,128,64)

norm_1, shape = build_normal_cell(inp, shape)

norm_2, shape = build_normal_cell(norm_1, shape)

merged = make_rc([norm_1,norm_2],'add')

reduce, shape = build_reduce_cell(merged, shape)

Listing 6-25

Stacking convolutional normal and reduction cells together in nonlinear fashion with residual connections

Cells can similarly be stacked in DenseNet style and with parallel branches for more complex representation. If you’re using linear cell designs, it’s recommended to use nonlinear stacking methods to add nonlinearity into the general network topology.

Nonlinear Cell Design

Nonlinear cell designs have only one input and output, but use nonlinear topologies to develop multiple parallel representations and processing mechanisms, which are combined into an output. Nonlinear cell designs are generally more successful (and hence more popular) because they allow for the construction of more powerful cells.

Building nonlinear cell designs is relatively simple with knowledge of nonlinear representation and sequential cell design. The design of nonlinear cells closely follows the previous discussion on nonlinear and parallel representation. Nonlinear cells form organized, highly efficient feature extraction modules that can be chained together to successively process information in a powerful, easily scalable manner.

Branch-based design is especially powerful in the design of nonlinear cell architectures. It is able to effectively extract and merge different parallel representations of data in one compact, stackable cell.

Let’s build a normal cell for image data that keeps the spatial dimensions and depth of the data constant (Listing 6-26, Figure 6-35). Like previous designs, it will take in both the input layer to attach the cell to, as well as the shape of the data. The depth of the image will be extracted from the shape and used as the depth throughout the cell. By using appropriate padding, the spatial dimensions of the data remain unchanged too. Three branches each extract and process features in parallel with different filter sizes; these representations are then merged via concatenation (depth-wise, meaning that they are “stacked together”). This merging produces data of shape (h, w, d · 3), as we are stacking together the outputs of three branches; to ensure that the output shape is identical to the input shape, we add another convolutional layer with filter (1, 1) to collapse the number of channels from d · 3 to d.

def build_normal_cell(inp_layer, shape):

    h,w,d = shape

    branch1a = L.Conv2D(d,(5,5),padding='same')(inp_layer)

    branch1b = L.Conv2D(d,(3,3),padding='same')(branch1a)

    branch1c = L.Conv2D(d,(1,1))(branch1b)

    branch2a = L.Conv2D(d,(3,3),padding='same')(inp_layer)

    branch2b = L.Conv2D(d,(3,3),padding='same')(branch2a)

    branch3a = L.Conv2D(d,(3,3),padding='same')(inp_layer)

    branch3b = L.Conv2D(d,(1,1))(branch3a)

    merge = L.Concatenate()([branch1c, branch2b, branch3b])

    out = L.Conv2D(d, (1,1))(merge)

    return out, shape

Listing 6-26

Build a convolutional nonlinear normal cell

We can similarly build a reduction cell by building multiple branches that reduce the spatial dimension of the input (Listing 6-27, Figure 6-36). In this case, one branch performs a convolution with stride (2, 2) – halving the spatial dimension. Another uses a standard max pooling reduction. Both are followed by a convolution layer with filter (1,1) that separately process the output before they are merged together via depth-wise concatenation. When the two are concatenated, the depth is doubled. This is fine in this case; since we want to compensate for decreases in resolution with corresponding increase in quantity of filters, there is no need to decrease the number of filters after concatenation as performed in the design of the normal cell. Correspondingly, we place a convolutional layer with filter (1,1) to further process the results of concatenation and use that layer as the output. The new shape of the data is correspondingly calculated and passed as a second output to the cell-building function.

def build_reduction_cell(inp_layer, shape):

    h,w,d = shape

    branch1a = L.Conv2D(d,(3,3), strides=(2,2), padding=’same’)(inp_layer)

    branch1b = L.Conv2D(d,(1,1))(branch1a)

    branch2a = L.MaxPooling2D((2,2),padding='same')(inp_layer)

    branch2b = L.Conv2D(d,(1,1))(branch2a)

    merge = L.Concatenate()([branch1b, branch2b])

    out = L.Conv2D(d*2, (1,1))(merge)

    new_shape = (np.ceil(h/2), np.ceil(w/2), d*2)

    return out, new_shape

Listing 6-27

Building a convolutional nonlinear reduction cell

These two cells (and any other additional designs for normal or reduction cells) can be stacked linearly. Because nonlinearly designed cells contain sufficient nonlinearity, there is less of a need to be aggressive in nonlinear stacking. Stacking nonlinear cells sequentially is a tried-and-true formula. One concern that could arise with linear stacking arises if you are stacking so many cells together that the depth of the network poses problems for cross-network information flow. Using residual connections across cells can help to address this problem. In the case study for this section, on the famed InceptionV3 model, we will explore a concrete example of successful nonlinear cell-based architectural design.

Case Study: InceptionV3

The famous InceptionV3 architecture , part of the Inception family of models that has become a pillar of image recognition, was proposed by Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna in the 2015 paper “Rethinking the Inception Architecture for Computer Vision.”² The InceptionV3 architecture, in many ways, laid out the key principles of convolutional neural network design for the following years to come. The aspect most relevant for this context is its cell-based design.

The InceptionV3 model attempted to improve upon the designs of the previous InceptionV2 and original Inception models. The original Inception model consisted of a series of repeated cells (referred to in the paper as “modules”) that followed a multi-branch nonlinear architecture (Figure 6-37). Four branches stem from the input to the module; two branches consist of a 1x1 convolution followed by a larger convolution, one branch is defined as a pooling operation followed by a 1x1 convolution, and another is just a 1x1 convolution. Padding is provided on all operations in these modules such that the size of the filters is kept the same such that the results of the parallel branch representations can be concatenated depth-wise back together.

A key architectural change in the InceptionV3 module designs is the factorization of large filter sizes like 5x5 into a combination of smaller filter sizes. For instance, the effect of a 5x5 filter can be “factored” into a series of two 3x3 filters; a 5x5 filter applied on a feature map (with no padding) yields the same output shape as two 3x3 filters: (w-4, h-4, d). Similarly, a 7x7 filter can be “factored” into three 3x3 filters. Szegedy et al. note that this factorization does not decrease representative power while promoting faster learning. This module will be termed the symmetric factorization module , although in implementation within the context of the InceptionV3 architecture it is referred to as Module A .

In fact, even 3x3 and 2x2 filters can be factorized into sequences of convolutions with smaller filter sizes. An n by n convolution can be represented as a 1 by n convolution followed by an n by 1 convolution (or vice versa). Convolutions with kernel height and widths that are different lengths are known as asymmetric convolutions and can be valuable fine-grained feature detectors (Figure 6-38). In the InceptionV3 module architecture, n was chosen to be 7. This module will be termed the asymmetric factorization module (also known as Module B ). Szegedy et al. find that this module performs poorly on early layers but works well on medium-sized feature maps; it is correspondingly placed after symmetric factorization modules in the InceptionV3 cell stack.

For extremely coarse (i.e., small-sized) inputs, a different module with expanded filter bank outputs is used. This model architecture encourages the development of high-dimensional representations by using a tree-like topology – the two left branches in the symmetric factorization module are further “split” into “child nodes,” which are concatenated along with the outputs of the other branches at the end of the filter (Figure 6-39). This type of module is placed at the end of the InceptionV3 architecture to handle feature maps when they have become small spatially. This module will be termed the expanded filter bank module (or Module C ).

Another reduction-style Inception module is designed to efficiently reduce the size of the filters (Figure 6-40). The reduction-style module uses three parallel branches; two use convolutions with a stride of 2 and the other uses a pooling operation. These three branches produce the same output shapes, which can be concatenated depth-wise. Note that Inception modules are designed such that a decrease in size is correspondingly counteracted with an increase in the number of filters.

Another often unnoticed but important feature of the Inception family of architectures is the 1x1 convolution, which is present in every Inception cell design, often as the most frequently occurring element in the architecture. As discussed in previous chapters, using convolutions with kernel size (1,1) is convenient in terms of building architectures like autoencoders when there is a need to collapse the number of channels. In terms of model performance, though, 1x1 convolutions serve a key purpose in the Inception architecture: computing cheap filter reductions before expensive, larger kernels are applied to feature map representations. For instance, suppose at some location in the architecture 256 filters are passed into a 1x1 convolutional layer; the 1x1 convolutional layer can reduce the number of filters to 64 or even 16 by learning the optional combination of values for each pixel from all 256 filters. Because the 1x1 kernel does not incorporate any spatial information (i.e., it doesn’t take into account pixels next to one another), it is cheap to compute. Moreover, it isolates the most important features for the following larger (and thus more expensive) convolution operations that incorporate spatial information.

The full InceptionV3 architecture is available at keras.applications.InceptionV3 with available ImageNet weights for transfer learning or just as a powerful architecture (used with random weight initialization) for image recognition and modeling.

Building an InceptionV3 module itself is pretty simple, and because the design of each cell is relatively small, there is no need to automate its construction. We can build four branches in parallel to one another, which are concatenated. Note that we specify strides=(1,1) in addition to padding='same' in the max pooling layer to keep the input and output layers the same. If we only specify the latter, the strides argument is set to the entered pool size. These cells can then be stacked alongside other cells in a sequential format to form an InceptionV3-style architecture (Listing 6-28, Figure 6-41).

def build_iv3_module_a(inp, shape):

    w, h, d = shape

    branch1a = L.Conv2D(d, (1,1))(inp)

    branch1b = L.Conv2D(d, (3,3), padding='same')(branch1a)

    branch1c = L.Conv2D(d, (3,3), padding='same')(branch1b)

    branch2a = L.Conv2D(d, (1,1))(inp)

    branch2b = L.Conv2D(d, (3,3), padding='same')(branch2a)

    branch3a = L.MaxPooling2D((2,2), strides=(1, 1),

                              padding='same')(inp)

    branch3b = L.Conv2D(d, (1,1), padding='same')(branch3a)

    branch4a = L.Conv2D(d, (1,1))(inp)

    concat = L.Concatenate()([branch1c, branch2b,

                              branch3b, branch4a])

    return concat, shape

Listing 6-28

Building a simple InceptionV3 Module A architecture

Besides getting to work with large neural network architectures directly, another benefit of implementing these sorts of architectures from scratch is customizability. You can insert your own cell designs, add nonlinearities across cells (which InceptionV3 does not implement by default), or increase or decrease how many cells you stack to adjust the network depth. Moreover, cell-based structures are incredibly simple and quick to implement, so this comes at little cost.

Input Shape Adaptable Design

One method of scaling is to adapt the neural network to the input shape size. This is a direct method of scaling the architecture in relation to the complexity of the dataset, in which the relative resolution of the input shape is used to approximate the level of predictive power that needs to be allocated to process it adequately. Of course, it is not always true that the size of the input shape is indicative of the dataset’s complexity to model. The primary idea of input shape adaptable design is that a larger input shape indicates more complexity and therefore requires more predictive power relative to a smaller input (the opposite relationship being true as well). Thus, a successful input shape adaptable design is able to modify the network in accordance to chances in the input shape.

This sort of scaling is a practical one, given that input shape is a key vital component of using and transferring model architectures (i.e., if it is not correctly set up, the code will not run). It allows you to directly build model architectures that respond to different resolutions, sizes, vocabulary lengths, etc. with correspondingly more or less predictive resource allocation.

This sort of resizing design has the benefit of portability in deployment; it’s easy to make valid predictions by passing in an image (or some other data form with appropriate code modifications) of any size into the network without any preprocessing. However, it doesn’t quite count as scaling in that computational resources are not being allocated to adapt to the input shape. If we pass an image with a very high resolution – (1024, 1024, 3), for instance – into the network, it will lose a tremendous level of information by naïvely resizing to a certain height or width.

This model, albeit simple, is now capable of being transferred to datasets of different input sizes.

We can similarly apply these ideas to convolutional neural networks. Note that convolutional layers can accept arbitrarily sized inputs, but nevertheless we can change the number of filters used in each layer by finding and applying generalized rules in the existing architecture to be scaled. In convolutional neural networks, we want to expand the number of filters to compensate for decreases in image resolution. Thus, the number of filters should increase over time relative to the original input shape to perform this resource compensation properly.

We can define the number of filters we want the first convolutional layer and the last convolutional layer to have as dependent on the resolution of the input shape (Listing 6-32). In this case, we define it the number of filters as , where w is the original width (the expression is purposely left in unsimplified form). Thus, a 128x128 image would begin with 8 filters, whereas a 256x256 image would begin with 16 filters. The number of filters a network has is scaled up or down relative to the size of the input image. We define the number of filters the last convolutional layer has simply to be 2³ = 8 times the number of original filters.

inp_shape = (128,128,3)

num_layers = 10

w, h, d = inp_shape

start_num_filters = 2**(round(np.log2(w))-4)

end_num_filters = 8*start_num_filters

Listing 6-32

Setting important parameters for convolutional scalable models

The sequence of filters for a (128,128,3) image, according to this method, is [8, 8, 8, 16, 16, 32, 32, 32, 64, 64]. The sequence of filters for a (256,256,3) image is [16, 16, 16, 32, 32, 64, 64, 64, 128, 128]. Note that we don’t define the architectural policy in this case recursively.

By determining which segment to enter by measuring progress rather than the deterministic layer (i.e., if the current layer is past Layer 3), this script is also scalable across depth. By adjusting the num_layers parameter, you can “shrink” or “stretch” this sequence across any desired depth. Because higher resolutions generally warrant longer depths, you can also generalize the num_layers parameter as a function of the input resolution, like num_layers = round(np.log2(128)*3). Note that in this case we are using the logarithm to prevent the scaling algorithm from constructing networks with depths that are too high in response to high-resolution inputs. The list and length of filters can then be used to automatically construct a neural network architecture scaled appropriately depending on the image.

Adaptation to input shapes is most relevant for architectures like autoencoders, in which the input shape is a key influence throughout the architecture. We’ll combine cell-based design and knowledge of autoencoders discussed in Chapter 3 to create a scalable autoencoder whose length and filter sizes can automatically be scaled to whatever input size it is being applied to.

We must consider the two key dimensions of scale: width and depth.

The number of filters in a block will be doubled if the resolution is halved, and it will be halved if the resolution is doubled. This sort of relationship ensures approximate representation equilibrium throughout the network – we do not create representation bottlenecks that are too severe, a principle of network design outlined by Szegedy et al. in the Inception architecture (see case study for section on cell-based design). Assume that this network is used for a purpose like pretraining or denoising in which bottlenecks can be built less liberally in size (in other contexts, we would want to build more severe representation bottlenecks). We can keep track of this relationship by increasing i by 1 after an encoder cell is attached and decreasing it by 1 after a decoder cell is attached.

In this example, we will build our neural network not with a certain pre-specified depth, but with whatever depth is necessary to obtain a certain bottleneck size. That is, we will continue stacking encoder blocks until the data shape is equal to (or falls below) a certain desired bottleneck size. At this point, we will stack decoder blocks to progressively increase the size of the data until it reaches its original size.

The complete model can be aggregated as ae = keras.models.Model(inputs=inp, outputs=x). This simple autoencoder design – using two convolutions followed by a pooling operation – has been scaled to be capable of modeling any input size resolution (as long as it is a power of 2, since pooling makes approximations that are not captured in upsampling if the side length is not cleanly divisible by the pooling factor – see Chapter 3 for more on this). Scaling a model to be adaptable to different input sizes can require more work, as we’ve seen, but it makes your model more accessible and agile in experimentation and deployment.

Parametrization of Network Dimensions

In the earlier section, we focused on scaling oriented toward a necessity-level parameter: the input shape of a model. In this section, we will discuss broad parametrization of network dimensions. Adapting the network architecture to the input shape requires us to generalize the model architecture and thus formulate implicit and explicit architectural policies that may or may not be successful. The goal here is rather to parametrize the dimensions of the network for the purposes both of adaptation to different problems and also for experimentation.

As discussed in the introduction to this chapter, seldom will one round of model building suffice for deployment. By parameterizing the dimensions of a network architecture, we are able to experiment with different scales and sizes more easily and quickly for a network to optimally fit a dataset.

The key difference between parametrizing a model for the sake of experimentation and inherent scalability and parametrizing a model to adapt to the input shape is that the factors determining parametrization are user-specified, not dependent on the input shape. Rather than programming architectural policies (e.g., the pattern with which the width of a network expands from the input shape), we use multiplying coefficients. These are parameters that the user specifies that are multiplied to the current dimensions of the network. A multiplying coefficient smaller than 1 will shrink that dimension, whereas a multiplying coefficient larger than 1 will expand that dimension.

In this case, we pass 2 into d() because 2 is the default number of layers in our model architecture. Note additionally that we are not scaling the entire depth of the network; we are leaving layers like batch normalization alone regardless of the depth coefficient, for instance. Depth-wise scaling should be applied appropriately to processing layers, not to layers like batch normalization or dropout that only shift or regularize the data flow.

The user can now adjust width_coef and depth_coef for quick experimentation and portability. The method by which you optimize the parametrization of network dimensions is up to you. One method that is likely to be successful is to use Bayesian optimization via Hyperopt or Hyperas to tune the width and depth scaling factors width_coef and depth_coef. Alternatively, one can look toward a recently growing body of research around general best practices for scaling , like the compound scaling method introduced in the successful EfficientNet architecture of models. We’ll explore this method in the case study for this section.

Similarly, you can parametrize the dimensions of nonlinear architectures without a nonlinear backbone that are simple, like parallel branches. If an architecture is too nonlinear to use a block-based approach toward scaling the depth dimension as introduced earlier, another method is to group these highly nonlinear topologies into blocks that can be scaled by stacking different quantities of blocks together.

By parameterizing network dimensions, you enable yourself and others to experiment and adapt the network architecture more quickly and easily, leading to improved performance on the problem.

Case Study: EfficientNet

Convolutional neural networks have historically been scaled relatively arbitrarily, along the two previously discussed dimensions – height and width – as well as (recently) resolution. “Arbitrary” scaling entails adjusting these dimensions of a network without much of a justification for how the adjusting is performed; there is ambiguity in how large to scale dimensions. The “larger is better” paradigm that gripped much of earlier development in convolutional neural network designs is reaching a limit in its competitiveness against other approaches that focus more on developing efficient mechanisms and designs. Thus, there is a need for a systematic method of scaling neural network architectures across several dimensions for the highest expected success (Figure 6-43).

Mingxing Tan and Quoc V. Le propose the compound scaling method in their paper “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.”³ The compound scaling method is a simple but successful scaling method in which each dimension is scaled by a constant ratio.

A set of fixed scaling constants is used to uniform scale the width, depth, and resolution used by a neural network architecture. These constants – α, β, γ – are scaled by a compound coefficient, ϕ, such that the depth is d = α^ϕ, the width is w = β^ϕ, and the resolution is r = γ^ϕ. ϕ is defined by the user, depending on how many computational resources/predictive power they are willing to allocate toward a particular problem.

Tan and Le propose explanations for the success of compound scaling that are similar to our previous intuition developed when adapting the architecture of a neural network based on the input size. By intuition, when the input image is larger, all dimensions – not just one – need to be correspondingly increased to accommodate the increase in information. Greater depth is required to process the increased layers of complexity, and greater width is needed to capture the greater quantity of information. Tan and Le’s work is novel in expressing the relationship between the network dimensions quantitatively.

Tan and Le’s paper proposes the EfficientNet family of models, which is a family of differently sized models built from the compound scaling method. There are eight models in the EfficientNet family – EfficientNetB0, EfficientNetB1, …, to EfficientNetB7, ordered from smallest to largest. The EfficientNetB0 architecture was discovered via Neural Architecture Search. In order to ensure that the derived model optimized both performance and FLOPS, the objective of the search was not merely to maximize the accuracy but to maximize a combination of performance and the FLOPS. The resulting architecture is then scaled using different values of ϕ to form the other seven EfficientNet models.

The EfficientNet family of models impressively obtains higher performance on benchmark datasets like ImageNet, CIFAR-100, Flowers, and others than similarly sized models – both manually designed and NAS-discovered architectures (Figure 6-44). While the core EfficientNetB0 model was created as a product of Neural Architecture Search, the remaining members of the EfficientNet family are constructed via scaling.

The EfficientNet model family is available in Keras applications at keras.applications.EfficientNetBx (substituting x for any number from 0 to 7). The EfficientNet implementation in Keras applications ranges from 29 MB (B0) to 256 MB (B7) in size and from 5,330,571 parameters (B0) to 66,658,687 parameters (B7). Note that the input shapes for different members of the EfficientNet family are different. EfficientNetB0 expects images of spatial dimension (224, 224); B4 expects (380, 380); B7 expects (600, 600). You can find this information in the expected input shape listed in Keras/TensorFlow applications documentation.

Looking at the EfficientNet source code in Keras is a valuable way to get a feel for how scaling is implemented on a professional level. It is available at https://github.com/keras-team/keras-applications/blob/master/keras_applications/efficientnet.py. Because this implementation is written for one of the most widely used deep learning libraries, much of the relevant code is used to generalize/parametrize the model for accessibility across various platforms and purposes. Nevertheless, its general structure of organization can be emulated for your deep learning purposes and designs.

The key EfficientNet() function defines two functions within itself, round_filters and round_repeats, which take in the “default” number of filters and number of repeats and scale them appropriately depending on the provided width coefficient and depth coefficient.

The round_filters function (Listing 6-42) takes in the default number of filters and returns the new number of filters after scaling. The equation for the new number of filters f_n is given roughly by , where d is the depth divisor, w is the width scaling coefficient, and f_o is the original number of filters. The number of new filters can never be shrunk below the value of the divisor because of the max(…) mechanism. The expression on the right simply multiplies the original number of filters by the width scaling coefficient and then applies the depth divisor to it. The depth divisor can be thought of as the bin size in quantization from Chapter 5; it is the basic unit that a parameter is scaled in terms of. The default divisor for EfficientNet is 8, meaning that the width is represented in multiples of 8. This sort of “quantization” can be easily done via the (integer division is performed in Python by a//d). This implementation adds to “balance” the scaled number of filters before “quantization”/“rounding.”

def round_filters(filters, divisor=depth_divisor):

    filters *= width_coefficient

    new_filters = max(divisor, int(filters + divisor / 2) // divisor * divisor)

    if new_filters < 0.9 * filters:

        new_filters += divisor

    return int(new_filters)

Listing 6-42

Keras EfficientNet implementation of the function used to return the scaled width of a layer

These functions are used in the building of the parametrized EfficientNet base model, allowing for easy scaling.