Software Entropy : Causes of Unorganized Code

When a program grows, it becomes unorganized more easily. Keeping our programs clean becomes more important. But why do programs become unorganized in the first place? Why do we need to organize more and do more cleanup work when a program grows? As we saw in the preceding section, there are multiple possibilities to solve the same programming problem. Obviously, the number of possible implementations grows with increasing size. Suppose we divide a four-line program into functions. For simplicity, let us assume we can move the boundaries between functions freely. We could create four functions with one line each, or we could have one four-line function, two two-line functions, and three other combinations (see Figure 14-1). Now consider dividing an eight-line program into functions. With the code being twice as long, we could create eight one-line functions. We could create one eight-line function. We could create between one and four two-line functions. We could create any combination of sizes in between. The number of possibilities grows much faster than the number of lines. If we take into account not only functions but all kinds of program constructs (lists vs. dictionaries, for loops vs. list comprehensions, classes, modules, etc.), the number of possible implementations for the same problem becomes practically infinite.

Figure 14-1. Possibilities to structure a program into functions. Top row: a four-line program; all six possibilities are shown (four one-line functions, two two-line functions, etc.). Bottom row: an eight-line program; only five out of many possibilities are shown. With more rows, the number of possible structures grows exponentially.

Which of these possible implementations is the best? That depends on what our program does, the kind of input used, what libraries we are using, and the expectations of the users. We will call this briefly the context of your program. Now there is a nasty fact: When a program grows, the context will change. When we add new functionality, new types of input, new libraries, the context shifts gradually. When the context changes, code that seemed the best solution before becomes an inferior one. Changing context is one main reason why finding the ideal implementation is hard.

A second cause of unorganized code is that programs are written by humans. For instance, it is easier to write code that works than code that works and is neatly written. Programs are usually written in multiple sessions. It occurs that we start adding something to the program one day, but forget to finish it the next. Last but not least, time pressure causes code to be written quickly instead of cleanly. Keeping code clean requires high self-discipline by the programmer and is a constant battle against laziness, forgetfulness, haste, and other manifestations of human imperfection.

For both reasons, changing context and human imperfection, it is an intrinsic property of all programs that they become unorganized over time. This phenomenon of code unorganizing itself has been termed Software Entropy, borrowing the concept from the Second Law of Thermodynamics. In less glamorous terms the law says: Disorder grows by itself. Whatever the reasons are, it is us who have to clean up. We need to know how to recognize, clean up, or prevent unorganized code.

Readability

In a bigger program, we spend more time reading than writing code. Therefore it is crucial that the code is understandable. We can ask ourselves a very simple question: Do we understand the code when we read it? Code written by both beginners and advanced programmers can be more or less readable. Clearly, littering code with print statements and commented lines like in the first example makes it less readable. In the second example, playing code golf (solving the problem with as few keystrokes as possible) doesn’t help either. Readability starts with small things. Compare the line

if text_str.find('REMARK') == 0:

with the semantically identical line

if line.startswith('REMARK'):

The second expression is closer to English, more explicit, and thus more readable. Readability manifests itself in many aspects of the code: choosing descriptive variable names , code formatting, reasonable choice of data structures (imagine writing a program using only tuples), code modularization, and use of comments. Of course, experience matters when reading code. Knowing advanced language features and some libraries is part of the game. But many times it is necessary to look at a section of code and quickly figure out what it does and why it exists instead of manually tracing the code line by line. If you find it difficult to answer these questions without executing the code in your mind, readability needs to be improved.

Structural Weaknesses

There are two very common kinds of structural weaknesses: The first is lack of structure. A typical sign that structure is lacking are big blobs of code not containing any structure: functions with 100 lines and more, programs without any functions, and so on. In Python, it is ok to write programs without any functions. For instance, I frequently write short data analysis scripts without even thinking about functions. But above 100 lines, they become throwaway code very quickly. Above 100 lines, structuring is necessary, and the bigger the program, the more important it becomes.

The second structural weakness are pseudostructures, code that looks structured but in fact is another form of monolithic blob. A straightforward example are multiple for loops, if conditions, and other code blocks like in the following example:

for line in tilefile:
    if not line.startswith('REMARK'):
        try:
            columns = line.split('	')
            if len(columns) == 3:
                x = int(columns[1])
                y = int(columns[2])
                if 0 < x 100:
                    if 0 < y < 100:
                        r = Rect(...)
                        ...

As a rule of thumb, when you reach the fourth level of indentation in any Python program (everything before column 16 being whitespace), something is weird. Usually the code can be improved by restructuring it. Especially, if there are multiple nested for loops, performance may quickly suffer.

A less obvious example of pseudostructures is code split into multiple functions, but with responsibilities that are not clearly defined. Consider the following example:

def get_number():
    a = input("enter first number")
    return a

def calculate(a):
    b = input("enter second number")
    result = float(a) + (b)
    print("the result of the addition is:", end="")
    return result

def output(r):
    print("{}".format(r))

num = get_number()
result = calculate(num)
output(result)

In this example, input and output are partially performed by the calculate function. The boundaries between functions are hard to understand, and the code becomes harder to manage. This kind of structural weakness becomes very common when the code becomes longer. There are many other kinds of structural weaknesses.

Redundancy

Redundancy violates the Don’t Repeat Yourself principle (DRY) of programming. Commonly, redundancy emerges from copy-pasting code fragments. The most obvious form of redundancy are duplicate lines. For instance, in the first code example of this chapter, the continue statement occurs twice:

...
if text_str.find('REMARK') == 0:
    ...
    continue
...
continue

In this case, the first continue is redundant, because the second one will be executed. The second continue is redundant as well, because the loop terminates anyway. Redundant lines like these increase the size of the program and compromise readability . In this case, we can simply remove them. Another kind of redundancy is when blocks of code repeat with small variations. Sometimes, repetitive blocks emerge, because the programmer uses Ctrl-C + Ctrl-V as a programming tool; sometimes they evolve by themselves. In both cases, duplicate code blocks can be removed by moving the redundant lines into a separate function or class. Such reorganizations, called refactoring, can become complex operations in a bigger program. A more subtle form of redundancy is redundancy in data structures. Given that redundancy occurs on many levels and is sometimes hard to see, it is not surprising that redundancy is among the top reasons for defects in larger programs.

Design Weakness es

Program design is a more difficult aspect. Good program design leads to robust programs that are tolerant toward unusual inputs and have a well-defined range of accepted values and clear error messages when the input is wrong. In general, robust design prevents defects from creeping in. In the MazeRun game, there is at least one design weakness: the random maze generator sometimes creates inaccessible spots in the maze. This is not a problem at the moment, but it might become one in the future.

A second aspect of good design is extensibility: How easy or hard is it to add something new to the program without breaking existing functionality. The fewer places need to be changed in order to implement a simple feature, the more extensible the design is. Good design anticipates what kind of changes will occur in the future. When you would like to evaluate a design, ask yourself: How comfortable would you feel about changing the code? If there are regions you would prefer not to touch, the design might need improvement.

Taken together, code that is hard to read, unstructured, redundant or contains design flaw s may be considered unorganized (see Figure 14-2 ). Often, several of these symptoms occur together. You can find all four symptoms in the two code fragments for loading the tile coordinates. Now imagine similar symptoms in a 1000-line program that stretches over many screen pages—again, the problem grows with increasing size. But it does not help us to complain about badly organized code; we need to think about how to improve it. Therefore, we will look at a few Best Practices of cleaning up Python programs next.

Figure 14-2. “I don’t get it. The kitchen got dirty all by itself.” Software becomes messy over time.

Place import Statements Together

To understand a program, it is necessary to know which other modules it requires to work (its dependencies). In Python, dependencies are mainly reflected by import statements. The first functional unit of a Python program should therefore be a separate block with all import statements. We simply collect all import statements from our program and move them to the beginning of the file. This way, it is easy to see which components the code requires at one glance. It is worth importing only the Python objects that are really used. In our case, there is a single import statement, and we only use pygame.Rect. Our import block becomes:

from pygame import Rect

We separate the imports from any code that follows by an empty line.

Place Constants Together

After the import section is a good place for all constants. A constant is a variable whose value does not change during program execution. Typical constants are input and output file names, path variables, column labels, or scaling factors used in calculations. We will simply collect all these constants in a separate section after the import block. In Python, there are no technical means to make a variable constant; their values can always be overwritten. To make it easier to distinguish constants from variables that change their value, Python constants are by convention written in UPPER_CASE letters. We have two constants in the tile coordinate loader. First, there is the size of tiles in pixels we need for calculating rectangles. We will place it in a constant SIZE. Second, there is the file name "tiles.txt". This file name assumes the file is in the current directory. To make the program usable from different locations, we need to provide the full path. We could write

TILE_POSITION_FILE = '/home/krother/projects/maze_run/tiles.txt'

However, this will only work on my own computer, which makes the code very inflexible. A better alternative for file names is to use the expression os.path.dirname(__file__) to determine the location of the current Python module. We can then add our filename to the path with os.path.join. The complete import and constant sections of the program are now:

import os
from pygame import Rect

CONFIG_PATH = os.path.split(__file__)[0]
TILE_POSITION_FILE = os.path.join(CONFIG_PATH, 'tiles.txt')
SIZE = 32

Again, we separate the constants from other code blocks by one or two empty lines. In the initial messy code, the variable TILE_POSITIONS looks like a constant but is modified by the program. We change it to lower case for later use:

tile_positions = {}

As a program evolves, many constants will change. In a program below 1000 lines, such changes are often easy to accommodate by editing the code. But if the values of constants change every second time you run the program, it is time to move them to an input file, to create a command-line option using the argparse module or read a configuration file using the configparser module.

Remove Unnecessary Lines

In programming, lines we think are important at first, later may turn out to be not important at all. A frequent intuitive reaction is to think “maybe I will need them later” and leave the unnecessary code in. However, a program is not a warehouse! Unnecessary code needs to be culled rigorously. If you have illustrative code examples that you don’t want to lose, copy them to a separate file and create a separate git commit for it. In our example, we have many examples of unnecessary lines: print statements, commented lines, and the redundant continue statements mentioned previously. We can simply delete them (seven lines in total). The program becomes much more readable immediately! Now it becomes easier to notice that there is an redundant variable assignment:

data = text_str

The variables data and text_str are identical. We can get rid of the extra assignment. We might also recognize that the following if condition is a blind alley:

if text_str.find('REMARK') == 0:
    text_str = text_str.strip()

The modified variable text_str is not used afterward. We can therefore get rid of this code block and replace the following else by the opposite of the if statement. As a result, our procedure becomes a lot clearer than it was before:

tile_positions = {}
for text_str in open(TILE_POSITION_FILE).readlines():
    x = text_str[2]
    if text_str.find('REMARK') != 0:
        y = int(text_str[4])
        r = Rect(int(x)*32, int(y)*32, int(32), int(32))
        key = text_str.split()[0]
        tile_positions[key] = r
print(tile_positions)

After removing lines, it is a very good moment to verify that the program is still working. We have already cleaned up many issues in our code, but we are not done yet.

Choose Meaningful Variable Names

Well-chosen variable names have a huge impact on readability. As a general rule, names containing English words are better than acronyms. English words that describe meaning are better than words describing a variable type. Very short variable names are often fine if the variables themselves are short-lived (e.g., x and y in our example). We can improve the readability by replacing r with rect, key with name, and text_str with row. Table 14-1 contains a few examples of good and bad variable names.

It is worth rechecking variable names from time to time. Their meaning changes while you develop the code—an incomprehensible variable name is bad, but a misleading one is even worse. With cleaned-up variable names, defects usually become easier to find.

Idiomatic Python Code

There are a few minor improvements to be made. We can use Python idioms, short, precise expressions that are applicable in many situations. Here, I would like to give just two short examples. First, we can make the conditional expression more readable, as mentioned previously:

if not row.startswith('REMARK'):

Second, we can use the csv module to pick the columns of the file apart. This is less error-prone than parsing the file yourself. Also, using the with statement is a generally recommended way to open files (because they are closed automatically afterward):

with open(filename) as f:
    for row in csv.reader(f, delimiter='	'):

Finding the right idioms is difficult, and opinions on which idiom is the best differ. It requires a combination of knowledge and experience. There is neither a complete catalog of Python idioms nor a set if clear rules when to use them. The closest thing is the book Fluent Python by Luciano Ramalho (O’Reilly, 2015).

Extract Functions

Probably the most important refactoring is to divide code into well-chosen functions. There are different reasons to write functions in Python. Here we do it mainly to divide a longer piece of code into smaller chunks. To extract a function from existing code, we need to write a function definition and define input parameters and a return statement for the output. We indent the code in between and add a call to the function, and a docstring. For instance, we can extract a function for creating Rect objects from our code:

def get_rectangle(row):
    """Returns a pygame.Rect for a given tile"""            
    x = int(row[1])
    y = int(row[2])
    return Rect(x*SIZE, y*SIZE, SIZE, SIZE)

Creating a separate function for just three lines might seem overkill at first. You may object that the code inside get_rectangle is too simple. But this is exactly the point! We want simple code. First, simple code stays clean for a longer time when Software Entropy sets in; for instance, if our function needs to cover one or two special cases (and grows), the code still will be readable. Second, simple code is understandable by other people (colleagues, interns, supervisors, or our successor). Third, simple code is more reliable under pressure: when programmers are plagued by deadlines, nervous managers, and after-dark debugging sessions, simple code is their best friend. We call the get_rectangle function from a second function load_tile_positions that contains most of the remaining code:

def load_tile_positions(filename):
    """Returns a dictionary of positions {name: (x, y), ..} parsed from the file"""            
    tile_positions = {}
    with open(filename) as f:
        for row in csv.reader(f, delimiter='	'):
            name = row[0]
            if not name.startswith('REMARK'):
                rect = get_rectangle(row)
                tile_positions[name] = rect
return tile_positions

When you want to split your own program into functions, you first need to identify a coherent piece of code and then move it into a function. There are some typical functions that frequently occur in Python programs:

• reading input like data files or web pages

• parsing data (i.e., preparing data for an analysis)

• generating output such as writing a data file, printing results, or visualizing data

• calculations of any kind

• helper functions, code extracted from a bigger function to make it smaller

A reasonable function size when restructuring a program is 5–20 lines. If the function will be called from multiple places, it may even be shorter. Like modules, functions deserve a triple-quoted docstring right after the function definition. The documentation string should describe what the function is doing in human language (avoid Python terminology if possible).

Create a Simple Command-Line Interface

After dividing our code into functions, it is time to create a top-level interface for the program. This interface avoids code being executed accidentally (e.g., by an import). To create the interface, we group all remaining function calls at the end of the program and wrap them in a separate code block. By convention, it starts with a weird if statement:

if __name__ == '__main__':
    tile_positions = load_tile_positions(TILE_POSITION_FILE)
    print(tile_positions)

The if expression is a Python idiom that appears strange at first (especially if you have seen other programming languages). Expressed in human language it means: “Execute the following block of code if this file is started as the main Python program. If this file is imported as a module, do nothing.” The __main__ block helps us to avoid accidental execution of the code. We now can import the module from somewhere else:

from load_tiles import load_tile_positions
tiles = load_tile_positions(my_filename)

In this case nothing is printed because the __main__ block is not executed when importing. The second use of the __main__ block is that we can run load_tiles.py as a Python program:

python3 load_tiles.py

Now we see the output produced by the print statement and can check whether it matches our expectations. Having a __main__ block in our program serves as a general entry point. If our module is not meant to be executed directly, we can use the __main__ block for simple test code (the code for the first part of this book contains a few examples). If we are writing a program that is to be used as a command-line tool, using the argparse module instead of sys.argv is a Best Practice. In a Python project, the bin/ directory is a good place for the command-line front end.

Structuring Programs into Modules

We have created separate modules throughout the first chapters already. In this chapter, we have worked on a single module. Therefore, we will simply list a few Best Practices to keep in mind when developing your own modules:

• Modules shouldn’t become too big. Modules of 100-400 lines are of a good size; modules up to 1000 lines are tolerable, but I recommend splitting them up as soon as possible.

• Each module should have a clearly defined purpose. For instance, loading data, writing data, and doing a calculation are all separate purposes that justify having their own modules. Also, if your constant section becomes large, it may be worth placing it in a separate module.

• Creating a module is as simple as moving a piece of code to a new file and adding import statements in the original file.

• Avoid circular imports at all costs. Whenever you come across a relation like A needs B, but B needs A, it is worth thinking about a better structure. It is always possible to avoid circular imports. You can avoid the problem by keeping A and B together, but probably this will cause problems later.

• When importing your own modules, write explicit imports (avoid import *).

• Add a triple-quoted docstring on top of each module.

Decomposing a program into separate modules is one of the easiest ways to structure programs.

Warning Messages

At the top of the pylintoutput, we find a section with warning messages that refer to PEP8 violations. Each warning contains the line number the warning refers to. For the code attributed to an inexperienced Python developer, we get

C: 1, 0: Missing module docstring (missing-docstring)
C: 7, 0: Invalid constant name "tilefile" (invalid-name)

whereas the cleaned-up code results in

C: 18, 4: Invalid variable name "x" (invalid-name)
C: 19, 4: Invalid variable name "y" (invalid-name)
W: 25, 4: Redefining name 'tile_positions' from outer scope (line 36) (redefined-outer-name)
C: 26,27: Invalid variable name "f" (invalid-name)
C: 36, 4: Invalid constant name "tile_positions" (invalid-name)

All of these warnings point us to things that could be improved. Variable name s with one character are discouraged, as is using a variable with the same name inside and outside a function. We could start renaming our variables (make them longer) and constants (to uppercase characters). However we will restrain ourselves for a moment and scroll to the bottom of the output.

Code Score

At the end the pylint output we find a score for our code of up to 10 points:

Global evaluation
-----------------
Your code has been rated at 7.73/10

Working with pylint is sometimes very rewarding. When we start fixing PEP8 issues, we can rerun pylint and see our score improve. This makes the PEP8 standard a bit treacherous. You may have noticed that after cleaning up our code, we have more PEP8 warnings than in the previous, messy code. This tells us that the warnings and the score pylint produces do not represent bigger changes in the code well. Focusing too much on style conformity distracts from more important issue. A Best Practice is to use pylint to conform with the PEP8 style guidelines, but don’t try to push every Python file to a pylint score of 10.0. Usually a score around 7.0 is already good enough. It is OK to ignore warning messages you do not agree with. Use your reason. According to Python core developer Raymond Hettinger, “PEP8 is a guideline, not a lawbook.” Think of PEP8 as a layer of paint on our building (see Figure 14-3). It improves how our code looks, but it does not support the roof.

Figure 14-3. Adhering to the PEP8 coding standard is like a good layer of paint: it looks beautiful and protects your code from bad weather.

Make It Work

Here, work means that the program finishes without Exceptions and that there are no semantic errors that we know of. In Part 1, we have already learned many debugging techniques to make a program work. In Part 2, we used automated testing to detect defects more thoroughly.

Make It Right

Making it right generally means organizing your code. In this chapter, we have already seen cleanup steps to make a program more readable and well-structured and to make the logic of execution transparent. However, these cleanup strategies are only the beginning. Keeping code well-organized becomes more important, as our programs grow further. Besides organizing functions and modules, designing classes and their interplay, building Python packages, and developing an architectureincluding all components of a system are topics where you can expect to find a lot of refactoring. These topics go beyond the scope of this book, though.

Make It Fast

When your program works correctly and is well-structured and readable, it is worth looking at its performance. Often at this stage, a program turns out to be fast enough already. If it is not, well-organized code is at least easier to tune for higher performance. There are many options to accelerate Python programs, ranging from adding computing power and compiling Python to faster programming languages and eliminating bottlenecks from the Python code itself. Performance optimization is not a topic of this book, but in Chapter 11 you find an example for writing performance tests.