Rough First Draft: Scope Without Detail

Customers don’t know what all their requirements are at the start. They have to see the system first. They’ll see some things they definitely don’t like, and that will help them articulate what they do want. This has always been true, and it’s a cornerstone of the Agile approach.

But, at some high level, customers do know what they want, and that’s what you need to capture in the rough first draft of the requirements. Don’t know what the reports should look like? Then the reports section can say: “There will be reports.” Not planning on any documentation? Then say: “There will be no documentation.” That will probably cause the customer to say: “What do you mean no documentation? We need documentation!” See, customers really do know what they want—just not the details. The first draft corresponds to what they know they want, and it’s vague about what they’re vague about. You write this first draft at the start of the project, before any development begins. These aren’t just-in-time requirements; they’re up-front requirements.

For many of the sections, I just don’t believe that waiting until halfway through development and being able to try out a half-dozen interim releases are going to help the customer know the answers. I could come up with a weird scenario in which the customer can’t know whether a German localization will be needed until there’s been a chance to try out the system, but, really, nothing is going to happen down the road to help with this. It’s about marketing, and it has nothing to do with using the system hands-on. The same goes for external interfaces, billing, installation, documentation, conversion, and a few other sections. Those decisions can be made at the start and should be, because they have a huge effect on design and development.

On the other hand, anything related to user-interface design, reports, and processing should be specified only in the most general way at the start. For these matters, the requirements should evolve as the developers and the customer work together with a live, if incomplete, system as a laboratory.

What’s important at the start, and at all times throughout the project, is that the scope be delineated, so it’s clear how much of the problem the application is supposed to address and, just as important, what it won’t address. How it solves the problem it’s supposed to solve—the detail—should be specified later, either because not enough is known about it until later or because the developers don’t need the details until later, or both. If the developers don’t need the details until later, such as exactly what reports are needed and what they look like, it’s better to wait. Any guess at the details might change anyway, and it’s better to work things out collaboratively between the customer and the developers when both are highly motivated to work on that part of the system. Neither party is inclined to work on such details at the start, when there’s so much else to think about.

Following was my rough first-draft requirements document for the Conference on World Affairs (CWA) system:

1. Database: Entities are Persons, Panels, Topics, Venues, Donations, Houses, Trips.
2. CRUD: A web page for each entity consisting of a form with a field for each column.
3. Processing: None. (The application pretty much just pushes data from place to place, not actually doing much of anything with it).
4. Reports: Panelists, Alpha List, Stable List, Betty Sheet, Trips, Housing. More will be specified later. Samples have been supplied. (Alpha List, Stable List, and Betty Sheet are CWA terms; it’s not important what they mean).
5. External Interfaces: None. Totally standalone. There might be a report to get data to populate the online schedule on the CWA website, but that will just be a CSV file.
6. I18N and L10N: None; English only. (Although data itself can be Unicode).
7. Accessibility: None other than what the OS offers (e.g., bigger cursor).
8. User Administration: Administrator and user logins. No finer-grained restrictions.
9. Billing: None.
10. Browsers and Platform: Safari and Chrome on MacOS, Internet Explorer and Chrome on Windows. Latest browsers only; no effort will be expended to support very old browsers. Nothing special for mobile. If it works on an iPhone, fine; if not, tough luck.
11. Installation: None, other than the single production system run by the University of Colorado Managed Services department.
12. Capacity: Five-to-ten simultaneous users. Database must hold 80 years of data (CWA started in 1948), with 100 panelists and 200 panels per year. Up to ten thousand donors.
13. Documentation: None.
14. Training: No formal training, but developer will meet with users occasionally.
15. Support and Maintenance: Support via email. An occasional phone call is OK, too. System will be updated and enhanced in future years as needed.
16. Conversion: From existing Excel spreadsheets and FileMaker database. CWA office will extract data and email it to developer.
17. Use Cases: TBD (to be determined). (Missing in the first draft, although I had a pretty clear idea of how the system was supposed to be used and had lots of screenshots, reports, and notes from the system I was replacing).

Bingo! The requirements! Only took about a half-hour to write, and they’re complete in scope, with nothing left out. Well, except for the details: what the CRUD pages look like and what the reports are.

Actually what I said to the CWA people at the start was even less detailed than the 17-section requirements document: “The system will handle panels, panelists and other persons, and donations, and it will generate all the reports you’re used to having.” That was exactly what they wanted to hear.

With the requirements in hand, I jumped into development. I started with the database design, which is where you should always start. (Reminder: This chapter and the rest of the book are only about PHP/MySQL applications.) Then, to check out the database, I did the conversion. With some real data in the database, I started in on the reports, using samples from previous conferences as my guide. At that point I was pretty sure the database design was basically OK. It would have to be modified as development proceeded, which is always the case, but it was fundamentally sound.

I didn’t show any of the reports to the CWA office staff because they were identical to the samples the staff had supplied me. I wasn’t ready yet to learn about new reports staff members might need. With a database-centric design, reports can be added without affecting the rest of the system, so I wasn’t worried.

At this point I was ready for the CRUD pages. I took a guess about what might work. Every time I showed what I had to the users, they had suggestions for how to make it better. This was the most time-consuming part of development, and the part where the users had the most input. Eventually we came up with a set of CRUD pages that they were happy with, and we went with those, with some additional tweaking as they used the system.

The same thing for reports. As I said, I copied the reports from past years (produced mostly from Excel) exactly, and we added a half-dozen additional reports as the CWA staff came up with ideas. I had also put in a generalized SQL-based query facility, originally for my own use, but the assistant CWA coordinator liked it so much that she taught herself SQL and started using it. She developed a set of about a dozen canned queries that might have been handled by reports developed by me at her request, but there was no need for me to do anything. Maybe this level of initiative is unique to assistant coordinators at universities, but you should check. Your users might be more capable than you think.

Toward the end of the this chapter, I’ll call my approach, which I’ve been using since the late 1960s, Planned Agile. It’s agile enough (lower-case “a”), but with a planning/design phase before any development starts.

A Closer Look at the Requirements Sections

Here are some additional comments on the 17 requirements sections.

Logging Requirements Changes

Log every request for a requirements change in the issue tracking system I described in Chapter 1, the same system used to log bugs and other support matters. This procedure serves three purposes, at least.

1. Ensuring that the proposed change isn’t misplaced or ignored.
2. Putting it on the agenda for a status or planning meeting. (I organize all such meetings from the issues tracker).
3. Documenting the change in case anyone in the future wants to know why the schedule slipped.

I like to keep things very simple, as you already know. Just record the text the way it came to you (e.g., text of an e-mail or copied from a memo), a brief title (five to ten words), a unique ID (e.g., 1234, REQ-0123, or let the tracker generate one), the date, who it came from, its category (“requirement”), and its status (“proposed”). Then, if it gets approved, change its status to “approved” and write fresh text, if needed, to document exactly what was approved. Keep the original text intact. HESK, the issue tracker I use, doesn’t allow me to customize statuses, so I give new requirements a status of “in progress” and then change them to “resolved” if and when they get approved.

When and if the requirement ever gets scheduled for a particular version, fill in a version number field. Then the issue log becomes the definitive list of what’s in the release. Fixed bugs get the same treatment.

A big mistake is to define so many fields in the issue tracker that it becomes a burden to log everything. Don’t forget that your job is to implement the application, not to win an award for documenting how you did it.

Modifying the Requirements Document

Here, too, keep the bookkeeping down to a level where it’s easy to do it, to increase the chance that you’ll actually do it. (That’s the way I am, anyway.) I like to make it very easy to keep the requirements document up to date.

An important principle in database design, as I explain in Chapter 4, is that the same data should never appear twice, because it’s too easy for the copies to get out of synch. That applies here, too: you don’t want to have a copy of the issue in the requirements document when it’s already in the issue tracker. So, until you’re ready to revise the requirements document, just reference a new requirement by its ID. (This is like using a foreign key in a database.) When you do revise the document, the issue tracker’s copy is no longer the primary reference, and should be marked that way so you’ll know. I’ll go through all of this step by step in the remainder of this section.

To make it reasonably easy to read requirements that refer to issues by ID, it’s convenient to generate a report of approved requirements from the issue tracking system to accompany the requirements document. That still means a lot of going back and forth for readers. Much better would be a script that automatically inserts the text of the issue from the tracker into the requirements document. It’s probably possible to do this with a Visual Basic script running in Microsoft Word, although I haven’t tried that. It’s much more straightforward to do if you write the requirements document as plain text, as I’ll demonstrate.

Note that this doesn’t violate the one-copy rule, since the source document only has a reference to the issue. The combined document, with the issue text automatically inserted is for viewing only, not for reediting. It’s just a report.

For an example, consider the baseline requirement shown in Listing 2-1, shown as it might be typed into a text editor.

Listing 2-1.  Housing Report baseline requirement

Housing Report
 
One row for each participant or other person to be housed.
 
Columns: Name, Companion, Housing Committee Contact, Housers Names, Houser Street/ZIP, Houser Phone, Arrival Trip Details, Departure Trip Details, Days Here, Smoking OK, Pets OK, Participant Notes
 
See sample from last year for format and other details.

Suppose a requirement change is approved. It’s in the HESK issue tracker as Issue 1553, as shown in Figure 2-3.

When this change is approved, the requirement is edited to reference the issue by its ID, as shown in Listing 2-2.

Listing 2-2.  Housing Report baseline requirement with reference to Issue 1553

Housing Report
 
One row for each participant or other person to be housed.
 
Columns: Name, Companion, Housing Committee Contact, Housers Names, Houser Street/ZIP, Houser Phone, Arrival Trip Details, Departure Trip Details, Days Here, Smoking OK, Pets OK, Participant Notes
 
See sample from last year for format and other details.
 
{Issue 1553}

Now for the best part: since all the issues are in a MySQL database (one reason I chose HESK) and the requirements document is a text file, it’s easy to write a PHP program that combines the two, as shown in Listing 2-3.

Listing 2-3.  Inserting Issues into Requirements Document

define(DB_USER, "rochkind_hesk");
define(DB_PASSWORD, "...");
 
$pdo = new PDO('mysql:host=localhost;dbname=rochkind_hesk',
  DB_USER, DB_PASSWORD);
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
 
$s = file_get_contents("CWA-requirements.txt");
$s = str_replace("
", "<br>", $s);
while (preg_match('/^(.*){(w+) (d+)}(.*)$/s', $s, $m))
    $s = $m[1] . issue($m[2], $m[3]) . $m[4];
echo $s;
 
function issue($cmd, $n) {
    global $pdo;
 
    $stmt = $pdo->prepare("select id, subject, message from
      hesk_tickets where id = :id");
    $stmt->execute(array('id' => $n));
    if ($row = $stmt->fetch()) {
        if ($cmd == "Issue")
            return "
                <table border=1 cellspacing=0 cellpadding=10>
                <tr><td>
                <p><b>Issue {$row['id']}: {$row['subject']}</b>
                <p>{$row['message']}
                </table>
              ";
        else
            return "<br>Issue {$row['id']}: {$row['subject']}";
    }
    else
        return "<b>[Can't locate Issue $n]</b>";
}

The PHP program replaced “{Issue 1553}” with the data from the database, and the output is shown in Figure 2-4, which is much easier to read.

As an advanced PHP programmer, you should be able to see what this program is doing, but here’s a very brief run-through anyway: the lines that assign to or reference the variable $pdo open a PDO connection to the HESK database and, inside the function issue, fetch the ID, subject, and message for that issue. (I talk much more about PDO in Chapter 5.)

The call to file_get_contents reads the entire text from the requirements document itself, and the next line puts in HTML brk tags to maintain the paragraphing. Next comes a preg_match loop to replace issue references (e.g., “{Issue 1553}”) with HTML that incorporates the issue details, supplied by the call to the function issue. Then it outputs the processed text.

The function queries the database for the issue data and formats the data as a table, if the full issue is wanted, or just the ID and subject otherwise. (I’ll explain the latter shortly.)

If plain text is too plain for you, which it is for me, you can use Markdown to add some formatting by augmenting the text with markers for headings, emboldening, and a few other embellishments. (See daringfireball.net/projects/markdown for the details.) Listing 2-4 shows the requirement with some Markdown added (### and **). Note that the document is still perfectly readable even with interspersed Markdown annotations.

Listing 2-4.  Housing Report requirement with Markdown added

### Housing Report
 
One row for each participant or other person to be housed.
 
**Columns:** Name, Companion, Housing Committee Contact, Housers Names, Houser Street/ZIP, Houser Phone, Arrival Trip Details, Departure Trip Details, Days Here, Smoking OK, Pets OK, Participant Notes
 
See sample from last year for format and other details.
 
{Full 1553}

There’s a free PHP implementation of Markdown you can use at michelf.ca/projects/php-markdown. (It’s also included in the downloadable source for this book at www.apress.com.) The changes to the PHP program to handle Markdown are trivial. The file has to be included at the top

require_once 'markdown.php';

and the two lines

$s = file_get_contents("CWA-requirements.txt");
$s = str_replace("
", "<br>", $s);

are changed to the single line

$s = Markdown(file_get_contents("CWA-requirements.txt"));

because Markdown automatically starts a new paragraph when a blank line appears. Figure 2-5 shows the new output, now decently formatted.

Even with the issues expanded inline within the requirements, the document can still turn into an unreadable patchwork if there are a lot of changes, as there surely will be. You’ll want to take time out at some point and produce another version that incorporates all the changes directly in the text. It’s still a good idea to reference the issues, however. A good way to do that is to use “Title” in the issue reference instead of “Issue” (emboldened), as in Listing 2-5, with the output shown in Figure 2-6.

Listing 2-5.  Updated Housing Report requirement.

### Housing Report
 
One row for each participant or other person to be housed.
 
**Columns:** Name, Companion, Housing Committee Contact, Housers Names, Houser Street/ZIP, Houser Phone, Arrival Trip Details, Departure Trip Details, Days Here, Smoking OK, Pets OK, Participant Notes
 
Where a companion has a different arrival and/or departure flight, show that info separately in the trip columns.
 
See sample from last year for format and other details.
 
*This requirement incorporates the following issues:*
{Title 1553}

As I said, once an issue is incorporated into a revision of the requirements document, its text in the issue tracker is for historical purposes only; the document is the authority. If the requirement has to be revised, a new issue has to be created. I added a custom field to HESK called “InDoc” to note issues that have been incorporated into the document and are therefore no longer the prime reference.

The small PHP utility shown here is a good example of how just a little bit of code can make a huge productivity difference. Imagine having to refer to the issue tracker, or even a report generated from it, for every issue reference. It’s not bad with only a few issues referenced from the requirements document, but in the real world, there are going to be more like a thousand. That would be intolerable.

What made it feasible for me to write this little program were two decisions about how I represented the requirements.

• The requirements document as a plain text file, augmented with Markdown.
• Requirements changes stored in a MySQL database, easily accessed from PHP.

All done with completely free software, and very little mechanism. As the architect Mies van der Rohe said, “Less is more.” Toss out Microsoft Word, free yourself from that expensive issue tracker with a proprietary database, and you’ve really got something.

Here’s another way to put it: all engineering documentation, including the requirements document, should also be able to be treated as data, and any databases should allow access from PHP (or some other scripting language). No proprietary formats allowed.

The Runaway Developer

Back in my days running XVT Software, a user-interface tool company I started in 1987, we needed a small subsystem that various modules could call to get parameter values. It’s the kind of thing that might be handled these days with XML, but, as was common back then, we invented our own simple property-value language for representing parameters (e.g., like Windows INI files or Mac OS plist files). With several highly educated computer scientists on board, we specified the language formally, leaving no wiggle room at all. Those were the complete, clear, and well-articulated requirements.

The developer assigned to the task of writing the code to read in and access the parameters went at it, but he was still working on his assignment after a couple of weeks. I figured a programmer with his talents should be able to write a little parser in at most two days, stick the parameters in a hash table or something like that, and be done with it. We were a small company and this was a very small part of the overall system. I myself have written code for similar tasks in less than a day.

So I called him in for a chat. He was very proud of his work, but he wasn’t quite done with it. I asked him to tell me how he had gone about it. It turned out that what he was doing wasn’t compiling our dumbed-down parameter language at all. He had defined a metalanguage for describing input specifications and was writing a compiler for that, along with an interpreter for the intermediate language he’d invented. Then, with this completed, all he had to do was write the specification for our language in his metalanguage, and, PRESTO!, he’d be done.

I’m embarrassed to say I lost my temper. I shouldn’t have, because it was my fault. He was only being himself.

The moral of this story? It’s quite enough to implement the requirements. A smart programmer might go a little beyond, to do some obvious generalization, such as allowing for five phone numbers if the requirement is for two. But inventing and implementing a new programming language when there was no requirement for any such thing, is going much too far. Enough of that kind of runaway development can kill a project.

(In the Agile world this is called YAGNI, for “you ain’t gonna need it,” but, when practiced literally, that’s going too far the other way. Sometimes several requirements can be absorbed into the same generalized facility.)

The Arzano Ranch

Mike, an acquaintance of mine, came to me and Ellen, a mutual friend, and asked us if we’d like to join him on a little programming project for an airline consultant named Ed Arzano. Ed had developed a new way of pricing seats on airplanes to maximize revenue based on some theoretical work done by an applied mathematician who was Mike’s brother. Very cozy.

I agreed and told Mike my hourly rate. He told me to charge a lot more, so I did. We all did.

The first meeting with Ed went swimmingly. He sketched out what he needed, and we all got on it right away. Mike worked on the algorithm, being mathematically trained, Ellen worked on a simple user interface, and I worked on the back-end data management. In a couple of weeks we had it done. We showed it to Ed, and he loved it. He took it off to show to an operations specialist at an airline, which I think was Frontier, based in Denver. They liked it, too.

Ed was so thrilled that when we next met with him he sketched out more ideas. We went off and revised the system. Same thing: We finished on time, Ed loved it, and off he went to demo his invention.

Next meeting with Ed, a rerun. More changes, more development, more demonstrations.

And on, and on, and on, for months. Ed was always happy, but never satisfied. Not because we were letting him down but because our system was inspiring him to come up with new ideas.

You’d think that months of never being done and having the requirements change every two weeks would be a bad thing, but it wasn’t. We enjoyed the work, and Ed was making real progress. He was a great guy, too. We were sorry to see the project end. (Years later I heard that he had started his own airline).

We made so much money on this gig that we bought a ski condo in Breckenridge, Colorado. The whole thing—not a timeshare. Of course, we named it the Arzano Ranch.

Moral: This all took place in the mid-1980s; if I’d thought of calling it Agile Development, I could have been somebody.

OK, a more serious moral: If the requirements are changing because reality is changing, and not because you didn’t bother documenting them, then go with the flow. Recall from Chapter 1 that satisfying the customer is what makes your project successful.