Hour 18. Test Early, Test Often

---

What You’ll Learn in This Hour:

• The wrong way (adding unit tests to an existing application)

• The right way (test-driven development and testing before code)

• How to use the available OCUnit testing macros to validate your application logic

• How to give the OCUnit framework a handle into your running app to test live application features

---

All the examples so far have focused on the aspects of Xcode necessary to complete the steps, and this has led to us displaying some fairly shoddy practices in terms of code development, testing, and debugging. Code littered with NSLog() calls, although convenient to use as an example (of something other than good code), is difficult to maintain and is quite unlikely to be well tested. Fortunately, Xcode integrates the SenTesting OCUnit Unit Testing framework, making it easy to use a well-principled testing methodology and enabling you to continually validate functionality in a manner that is much easier than reading NSLog() output.

The OCUnit Unit Testing framework, and all unit-testing frameworks in general, is intended to support a very specific testing paradigm that has emerged from recent advances in programming practices. This paradigm, known as test-driven development, is a software development method in which tests that validate functionality are written before the code that implements the functionality. If you’re not familiar with the idea, this might sound odd, or even a bit crazy, but done well, the result can be code that is much easier to maintain and much less time spent in the debugger trying to figure out why something stopped working. However, if you are like most people, you are coming at unit testing with some code already written and a need to implement testing into that existing code for quality control. The OCUnit framework can help with this goal, as well, but it is much harder to make sure that you have tests written to validate every aspect of your software if you start late in the game.

Adding Unit Tests to an Existing Application

How you go about adding unit tests to an existing application depends on whether you were thinking ahead back when you created the project for the application. If you indicated that unit tests should be included when you initially created the project, even though you haven’t been using them while coding, almost everything is set up, and you just need to start adding code to the unit-test target to start reaping the benefits of unit testing.

If you did not originally tell Xcode to include the Unit Test module, or if the project is coming from an earlier version of Xcode where unit testing wasn’t integrated, you have a bit more work to do. If you’re lucky enough to have created your project with tests enabled, skip ahead to the “Implementing Tests for Existing Code” section; otherwise, complete the following section before starting to implement tests. We use Apple’s OpenGL demo code, available from

https://developer.apple.com/library/mac/samplecode/GLEssentials/GLEssentials.zip, for this example. As delivered, the example’s code does not include the unit-testing framework or any instantiated tests.

To add unit tests to an existing application, follow these steps:

1. Open your existing unit-test-less project in Xcode. You can tell that it is test-less because it does not include any Tests group for files within the project nor any target.octest product in the Products group, as shown in Figure 18.1.

Figure 18.1. You can tell that this project is unit-test-less because it does not have any Tests group for files nor a target.octest product in the Products group.

2. Select the project in the Navigator.

3. Click the + (Add Target) icon at the bottom of the Xcode window.

4. Select the Other category of templates (for either OS X or iOS, as appropriate for your environment), select the Cocoa Unit Testing Bundle, as shown in Figure 18.2, and then click Next.

Figure 18.2. Selecting the Cocoa Unit Testing Bundle.

5. Enter a product name for your tests. It probably makes the most sense to give your testing suite for an application a name similar to the application. I named mine GLEssentials. A smarter name to pick might be GLEssentialsTests.

6. Enable Automatic Reference Counting. Make sure the Project is still set appropriately. Enter your company identifier or abbreviation and click Finish. As fully filled out, the target options should appear as shown in Figure 18.3.

Figure 18.3. Filling out the target options for the test suite.

After completing these steps, return to the Xcode window that appears in Figure 18.4. A new GLEssentials.octest product should exist in the Products group in the Navigator. A new GLEssentials group, containing an implementation file and a header file, should now appear in the Navigator, just above the Frameworks group. A new GLEssentials Target, with an icon that looks like a Lego block should appear under the Targets list in the Editor area. The SenTestingKit and Cocoa frameworks should have been added in the Frameworks group, below Linked Frameworks.

Figure 18.4. The Xcode interface just after adding our test suite.

You’re getting closer, but you still have more work to do. If you try to run tests right now, as shown in Figure 18.5, Xcode helpfully reminds you of what you still need to do.

Figure 18.5. When trying to run tests now, Xcode tells us that we must first configure the project for testing.

7. If you tried that, just to be contrary, click the Edit Scheme button. If you’re not the contrarian, under the Product menu, select Edit Scheme.

8. The Scheme Editor opens. Select the Test scheme, as shown in Figure 18.6. Configure it for testing the debug or release versions of the application, as needed.

Figure 18.6. Select the Test scheme and configure it for use with either the debug or release versions of the application.

9. Click the + button below the (empty) list of tests.

10. In the dialog that opens, open the project, and select the testing bundle you created, as shown in Figure 18.7. Then click Add.

Figure 18.7. Selecting the testing bundle created earlier.

11. Back in the Scheme Editor, you should now see your testing bundle. If you click its reveal triangle, you’ll see your testing class, with an additional reveal triangle, and if you open it, you’ll see a single test method. Each of these has a checked check box next to it, indicating that it is enabled for testing with the current scheme setting, as shown in Figure 18.8. Close the Scheme Editor by clicking OK.

Figure 18.8. Then testing bundle now appears in the Scheme Editor and includes a testing class with a single test.

12. Click the Run button in the upper-left corner of the Xcode interface and wait until a pop-up menu of run options appears. Select Test, or under the Product menu, select Test. A short time will pass. If you watch the build status, you’ll see reports of compiling code flashing by, and then your build will... fail.

But, this is a good thing. If you look in the Navigator, you’ll see your GLEssentials test class with a red error icon, as shown in Figure 18.9. Click its disclosure triangle to see the source of the error and you’ll see that it is something in your GLEssentials class implementation that produced the error. Reveal the details under the GLEssentials.m implementation file and you’ll see a bit more (cramped) detail. Click the red error icon beside the brief error details, and the source of the error will open in the Editor area. As you can see from Figure 18.10, the actual “failure” in the build is that the test case example (named testExample) successfully ran. And success, in the case of running the example method provided by Apple, is for it to throw an STFail message, complaining that you have not actually implemented any tests, which probably means it is time to add some.

Figure 18.9. In the Navigator panel, the red error icon is showing.

Figure 18.10. In the Editor area, we see that testExample ran but did not have any tests implemented.

Implementing Tests for Existing Code

Implementing tests for existing code usually sounds easier than following the test-driven development (TDD) mantra of implementing tests before functionality. In fact, though, it is usually rather difficult to figure out exactly what to test and how to test it. If you follow the TDD mantra and implement tests first, your code tends to be designed in modules that can be tested. If you implement code first and add tests later, it is quite often the case that the functionality that you want to test is buried in methods that because of their reliance on user interaction, network access, or other complicated behavior are quite difficult to test in an automated fashion.

If you are faced with this situation, you must put on your thinking cap and identify components within the application that can be tested and components that cannot. For those that can, you can implement tests as follows:

1. In the Navigator, select your testing class implementation file and open it in the source editor.

2. Delete Apple’s testExample method.

3. Create a new instance method with a void return type. Make sure its name starts with test. The rest of its name should reflect what it is going to test for you. For instance, let’s test Apple’s vector math addition routine, found in the vectorUtil.c source file. So, we name our test method testVec4Add. The bare method should look something like Listing 18.1.

Listing 18.1. Bare Method for Testing vectorUtil.c

---

- (void)testVec4Add
{
}

---

4. To get access to the method (vec4Add is a function in this case), we need to add its code to our test case target. So, select the vectorUtil.c file from the Other Sources group in the Navigator, open the Utilities, and select the testing bundle GLEssentials for Target Membership, in addition to the already selected OSXGLEssentials application target, as shown in Figure 18.11.

Figure 18.11. Setting up vectorUtil.c so that the test method can access it.

5. Up at the top of the implementation file, add an #import line for the vectorUtil.h file.

6. Now add code to test the Vec4Add routine to your testVec4Add testing method.

Exactly how you choose to test this method is up to you; in fact, if this were your project, you would probably want to test several different features of it, possibly with several different testing method implementations. For example, it is, of course, appropriate to test whether it properly adds two vectors and produces the correct mathematical result. It is also appropriate to test whether it handles null pointers, or vectors containing non-numeric values, or vectors with different storage sizes properly. Looking at Apple’s implementation, I’m guessing the answer is no.

To actually test the method (function, in this case) under consideration, you need to build the appropriate parameters to pass to the method (or function), invoke the method with the parameters, and then examine the result. The SenTesting suite provides a number of STAssert macros that perform the “examine the result” part of that process for you; you just have to pick the right one.

The primary macros you’ll want to choose from are probably from the highly useful subset of the complete bunch shown in Table 18.1.

Table 18.1. The Most Useful STAssert Macros for Testing Code Unit Functionality

Others, for more specialized uses, are defined and documented in /Developer/Library/Frameworks/SenTestingKit.framework/Versions/A/Headers/SenTestCase_Macros.h.

In each case, the macro takes a return value or values from your tests, compares them to expected values or each other, and if they don’t pass whatever variety of assertion the macro applies, the format_string (an NSString) is used to produce a printf-type formatted string from the remaining args... values, a failure exception is thrown for the build, and the formatted result is logged.

Putting this together for the Vec4Add function, and planning to test whether it properly adds two proper vectors, we come up with a testVec4Add method that looks something like Listing 18.2.

Listing 18.2. testVec4Add Method for Adding Two Proper Vectors

---

- (void)testVec4Add
{
    float *vecA, *vecB, *vecC, *vecD;
    int i;

         NSLog(@"Testing vec4Add");
    vecA = malloc(sizeof(float)*4);
    vecB = malloc(sizeof(float)*4);
    vecC = malloc(sizeof(float)*4);
    vecD = malloc(sizeof(float)*4);

    vecA[0] = 1.2; vecA[1] = 3.4; vecA[2] = 5.6; vecA[3] = 7.8;
    vecB[0] = 9.9; vecB[1] = 7.8; vecB[2] = 5.7; vecB[3] = 3.6;
 // vecC;
    vecD[0] = 11.1; vecD[1] = 11.2; vecD[2] = 11.3; vecD[3] = 11.4;

    vec4Add(vecC, vecA, vecB);

    for(i=0;i<4;i++)
    {
        STAssertEquals(vecC[i], vecD[i],
                       @"vec4Add index %d Exp %f Rec %f",
                       i, vecD[i], vecC[i]);
    }

    free(vecA); free(vecB); free(vecC); free(vecD);
         NSLog(@"Done Testing vec4Add");
}

---

Now, if you run your tests, the result will be yet another fail, as shown in Figure 18.12. This time, however, it is not such a good thing that it is a failure because the test is actually telling you that the vec4Add routine is producing incorrect results. Examined more closely, the specific errors are as follows:

Figure 18.12. The tests fail because vec4Add is producing incorrect results.

GLEssentials.m: error: testVec4Add (GLEssentials) failed: '11.099999' should be equal to '11.100000': vec4Add index 0 Exp 11.100000 Rec 11.099999
GLEssentials.m: error: testVec4Add (GLEssentials) failed: '11.200001' should be equal to '11.200000': vec4Add index 1 Exp 11.200000 Rec 11.200001
GLEssentials.m: error: testVec4Add (GLEssentials) failed: '11.299999' should be equal to '11.300000': vec4Add index 2 Exp 11.300000 Rec 11.299999

Because I added some helpful debugging code in the return message format string, I can see that the problem looks like a floating-point precision error. For example, we’re supposed to be receiving 11.1 back in the 0th vector component of the summed vector, and instead we’re receiving 11.099999 (and probably some more repeating 9s after that).

Chances are, this is not really an error but simply a limitation on the precision of the variables we are using. So, let’s try updating the STAssert macro we’re using, to compare the values with some tolerance for error. Changing the STAssertEquals macro invocation and replacing it with STAssertEqualsWithAccuracy, with the accuracy tolerance set at 0.000001, results in the long-hoped-for behavior of the testVec4Add method passing its test, as shown in Figure 18.13.

Figure 18.13. After you adjust the accuracy tolerance, the tests pass.

If you examine the Debug area, you’ll see that the SenTestingKit framework reports when it starts the test case testVec4Add, the NSLog lines I added to validate that my test was running appear, and then the SenTestingKit reports the successful conclusion of the test and the time it took to complete. Parsing the NSLog()-based output from the OCUnit testing module is not the easiest task in the world, but because it is tightly integrated into Xcode, failures come back to the GUI, and you really only have to look at the log if you want more information about your successes.

Finally, to verify that the test case actually catches real errors, we induce an actual error in the vec4Add routine. As delivered from Apple, the routine is as appears in Listing 18.4.

Listing 18.4. vec4Add Routine from Apple

---

void vec4Add(float* vec, const float* lhs, const float* rhs)
{
       vec[0] = lhs[0] + rhs[0];
       vec[1] = lhs[1] + rhs[1];
       vec[2] = lhs[2] + rhs[2];
       vec[3] = lhs[3] + rhs[3];
}

---

Edit it so that the third vector component assignment reads as follows:

vec[2] = lhs[1] + rhs[3];

Now run your test case again. You should first see a report that the build succeeded. Because the edit is legal C code, this is expected. Xcode should be able to build the application; it just shouldn’t run properly anymore. Immediately after, you should see a message that says “Tests Failed” and the dreaded red error icon appears in the Status field, and the Navigator switches to a view showing you the cause of the error. In this case, a received value of 7.0 should be equal to an asserted value of 11.3. Because we broke the vector adder, it did not correctly add 5.6 + 5.7, and instead added 3.4 + 3.6. The test fails (correctly) when the adder does not work. All is well, but fix the bug you introduced in the vec4Add function and retest it, just for good measure.

Unit testing the rest of the application logic proceeds from here in an iterative fashion, as follows:

1. Identify a method, function, or other component of your code that needs tested.

2. Figure out how to test it.

3. Add a test method to your testing class.

4. Add code to implement you test.

5. Tweak the code that implements the test until it passes for correct output from the application code.

6. Tweak the application code so that it produces incorrect output so that you can be sure the test properly fails.

7. Fix your application code again and retest.

8. Repeat as necessary.

If you find that you need to implement conceptually distinct subsets of unit tests—for example, a group that tests vector math functions, and a different group that tests image rendering, or a group that tests vector math with floats and a different group that tests it with integers—you can facilitate this by adding additional testing classes. This approach lets you segregate your testing methods so that they can be grouped with appropriate setup methods so that only the correct and necessary setup values and variables are created. It also facilitates enabling and disabling groups of related tests, in cases where you want to focus on specific features of the code and not waste time on regions where you either know that things are currently correct or know that they are currently broken but not a priority.

If you want to add additional test classes, you can do this by dragging test class templates out of the File Template Library in the Utilities, into the group for the testing bundle in the Navigator. To add a class, follow these steps:

1. Show the File Template Library in the Utilities and drag an Objective-C Test Case Class template and drop it into your Tests group in the Navigator, as shown in Figure 18.14.

Figure 18.14. Dragging a test class template into the project.

2. In the dialog that opens, shown in Figure 18.15, name your test class and optionally create a new directory for it. Do not click Create yet.

Figure 18.15. Provide a name and optionally create a new directory for the test class in the dialog that appears.

3. Deselect the application target from the targets and select the building-block iconed testing bundle that you created in the first steps of this hour. Now click Create. Back in the main Xcode interface, you should see that a new pair of files, one implementation and one header for your new class, has appeared in the group for your testing bundle.

Test your application again and examine the contents of the Debug area. You’ll see that a new test has been added and passed. This is the default testApp test method from the class you just added. You can add more test methods to this new class, and you can add setUp and tearDown methods copied from your original testing class to initialize specific items that the tests in this class need to function.

Accessing the Rest of an Application Through the Bundle Loader

In the preceding example, we followed the canonical unit-testing path of linking the source for the methods we want to test with the unit-testing class we’re implementing. This is clean, neat, and significantly reduces the running time and memory overhead for your tests, but it does not provide a way for you to test your methods in situ, only in the “clean room” environment of the unit-testing framework.

Fortunately, Xcode provides a convenient way to work around this limitation, in the form of the Bundle Loader, which, with only two tweaks to the build settings for your project, can give your unit-test classes full access to the application internals. Formally, many tests that you can write this way are not actually unit tests but are rather integration tests, but you will probably be happy with the functionality no matter what it is called.

To enable the Bundle Loader, follow these steps:

1. Select your project in the Navigator and your testing target from the Targets in the Editor area.

2. Open the Build Settings tab for your testing target.

3. Select All rules, Combined.

4. Enter the word bundle (no quotation marks required) into the Search field. The giant list of cryptic settings should condense down to what is shown in Figure 18.16.

Figure 18.16. Searching for bundle.

5. Double-click the Bundle Loader field. A pop-up editing field appears.

6. Enter $(BUILT_PRODUCTS_DIR)/OSXGLEssentials.app/Contents/MacOS/OSXGLEssentials into the editing field, and then click Done.

7. Cancel the search for bundle and search now for host. A setting for Test Host should appear.

8. Double-click the Test Host field. A pop-up editing field appears.

9. Enter $(BUNDLE_LOADER) into the editing field and click Done.

10. Select your application target (not your testing target) in the left column of the Editor area.

11. Open the Build Settings tab for it. Select All rules, Combined.

12. Enter the word symbols into the Search field.

13. Find the setting for Symbols Hidden by Default.

14. Click the (default) Yes setting and change it to No, as shown in Figure 18.17.

Figure 18.17. Changing the Symbols Hidden by Default setting from Yes to No.

Now you have access to the symbol space of your application. If you run your tests, you’ll probably notice that the application window briefly appears. With this setup, you can access methods from the application without explicitly linking specific source files with the testing target. Moreover, and more usefully, if you add appropriate getter and setter methods to expose application internals like the NSWindow pointer, you can send messages to the instantiated methods in the running application.

For example, if you use this technique to enable the Bundle Loader for Unit Tests in BeeLine (Hour 15, “Putting It All Together: Building an OS X Application”), you can add a test case with code like Listing 18.5, and it will automatically draw a list of points into the QuartzGrough view, just as though you entered the X and Y values and clicked the Add button yourself.

Listing 18.5. Test Case Code That Automatically Draws a List of Points into the QuartzGrough View

---

- (void)testPointAddingPipeline
{
    NSRect testRect;
    testRect.origin.x = 10;
    testRect.origin.y = 20;
    testRect.size.width = 200;
    testRect.size.height = 100;

    QuartzGrough *aGroughGraph = [SGFAppDelegate getTheGroughGraph];

    aGroughGraph.anX = 0.4;
    aGroughGraph.anY = 0.6;
    [aGroughGraph plotUpdate];

    aGroughGraph.anX = 0.6;
    aGroughGraph.anY = 0.8;
    [aGroughGraph plotUpdate];

    aGroughGraph.anX = -0.3;
    aGroughGraph.anY = 0.2;
    [aGroughGraph plotUpdate];

    [aGroughGraph drawRect:testRect];
}

---

If you want to experiment with BeeLine with the Bundle Loader enabled, and a test case configured to drive the interface, you can download the project from the hour 18 source code at http://teachyourselfxcode.com.

Summary

In this hour, you learned about unit testing and how to apply the SenTesting framework (OCUnit) to develop unit tests for your application logic. You learned how to add the testing framework to an existing application and why it is a better idea to start building unit tests immediately from the beginning of your project. You learned how to write test methods and how to make them correctly pass when they should pass and fail when they should fail. Finally, you learned how to access the running application bundle, enabling the use of the unit-testing framework for some types of integration testing, as well.

Q&A

Q. How could you handle unit-testing the plotPointx method of the BeeLine application delegate? It relies on user input and receiving UI events from a sender.

A. One possible answer is that unit testing of complex code often requires the implementation of components called mocks, fakes, and stubs. Each of these (and there is some disagreement in the community exactly what is meant by the different terms) is a variety of stand-in bits of code that in some fashion pretends to be an interface or a connection to a database or other complex connection. Used properly, they can pretend to be the UI and send event messages in to your application as though a user clicked an interface component. There are rapidly evolving libraries available on the Internet to aid you in developing mocks for your unit tests. Because they are changing so rapidly, we recommend an Internet search to find the most recent, but http://www.ocmock.org/ is probably a good place to start.

Another possible answer is that the difficulty in figuring out how to test this call should suggest to you that the implementation is less clean than might be desired. This is probably an opportunity to refactor the code and separate out the functionality of what to do with the results of a UI action and the receipt of the UI action itself.

Q. How in the world does the test-driven development paradigm of writing tests before you write code work?

A. Usually painfully, at first. The TDD paradigm requires that you be able to articulate what a method should do before you implement it. This is probably a good idea overall. It does not lend itself to organically growing applications, but it does drive a workflow where you know how to test every step of an implementation because you know what to expect from a method before you write it.

Q. What if I expect the wrong thing from the code?

A. That is one of the downsides to TDD. When the same person is writing the tests and the code, the tests and the code may be subject to the same blind spots. This is why debugging, which we cover in the next hour, is still required even when you are using TDD methods.

Workshop

Quiz

1. Why can you just start adding testing methods to some of your projects, invoking methods out of your application and frameworks without doing any further configuration, while other projects complain with linker errors claiming they cannot find the application methods?

2. How can you test whether two pointers point to the same memory location in a unit test?

3. What can you do if what is required for “passing” a particular test cannot be encoded into any of the STAssert comparisons?

Answers

1. Projects that were initially set up with unit testing turned on have the Bundle Loader configuration already set up and therefore do not require you to explicitly link the source files for the methods you want to test with the testing framework target.

2. STAssertEqual, using the values of the pointers, will accomplish this. Make sure you do not use the targets of the pointers because then STAssertEqual will not fail when you hand it different pointers that point to different variables that have identical values.

3. Write your test case to carry out whatever conditional evaluation you require, and invoke STFail() directly if your evaluation indicates test failure.

Activities

1. Implement unit tests for each of the BetterList framework methods. Exercise as many of the STAssert macros as you can.

2. Add a new method to the BetterList framework using the TDD “tests before code” paradigm. Make the new method roll the list down—that is, have it move the tail of the list before the head, making the former tail into the new head, the former head the second item, and the former next-to-last item into the tail. Implement the unit tests for this method before implementing the method, and then develop the method implementation to satisfy your tests. Test and develop until your tests pass.