IN THIS CHAPTER
Web Page Fundamentals
Black-Box Testing
Gray-Box Testing
White-Box Testing
Configuration and Compatibility Testing
Usability Testing
Introducing Automation
The testing techniques that you've learned in previous chapters have been fairly generic. They've been presented by using small programs such as Windows WordPad, Calculator, and Paint to demonstrate testing fundamentals and how to apply them. This final chapter of Part III, “Applying Your Testing Skills,” is geared toward testing a specific type of software—Internet web pages. It's a fairly timely topic, something that you're likely familiar with, and a good real-world example to apply the techniques that you've learned so far.
What you'll find in this chapter is that website testing encompasses many areas, including configuration testing, compatibility testing, usability testing, documentation testing, security, and, if the site is intended for a worldwide audience, localization testing. Of course, black-box, white-box, static, and dynamic testing are always a given.
This chapter isn't meant to be a complete how-to guide for testing Internet websites, but it will give you a straightforward practical example of testing something real and give you a good head start if your first job happens to be looking for bugs in someone's website.
Highlights of this chapter include
What fundamental parts of a web page need testing
What basic white-box and black-box techniques apply to web page testing
How configuration and compatibility testing apply
Why usability testing is the primary concern of web pages
How to use tools to help test your website
In the simplest terms, Internet web pages are just documents of text, pictures, sounds, video, and hyperlinks—much like the CD-ROM multimedia titles that were popular in the mid 1990s. As in those programs, web users can navigate from page to page by clicking hyperlinked text or pictures, searching for words or phrases, and viewing the information they find.
The Internet, though, has introduced two twists to the multimedia document concept that revolutionizes the technology:
Unlike data that is stored solely on a CD-ROM, web pages aren't constrained to a single PC. Users can link to and search worldwide across the entire Internet for information on any website.
Web page authoring isn't limited to programmers using expensive and technical tools. The average person can create a simple web page almost as easily as writing a letter in a word processor.
But, just as giving someone a paint brush doesn't make him an artist, giving someone the ability to create web pages doesn't make him an expert in multimedia publishing. Couple that with the technology explosion that continually adds new website features, and you have the perfect opportunity for a software tester.
Figure 14.1 shows a popular news website that demonstrates many of the possible web page features. A partial list of them includes
A great deal of functionality also isn't as obvious, features that make the website much more complex:
Customizable layout that allows users to change where information is positioned onscreen
Customizable content that allows users to select what news and information they want to see
Dynamic drop-down selection boxes
Dynamically changing text
Dynamic layout and optional information based on screen resolution
Compatibility with different web browsers, browser versions, and hardware and software platforms
Lots of hidden formatting, tagging, and embedded information that enhances the web page's usability
Granted, short of a secure e-commerce website, this is probably one of the more complex and feature-rich web pages on the Internet. If you have the tester mentality (and hopefully you've gained it by reading this far in the book), looking at such a web page should whet your appetite to jump in and start finding bugs. The remainder of this chapter will give you clues on where to look.
Remember all the way back to Chapters 4 through 7, the ones that covered the fundamentals of testing? In those vitally important chapters, you learned about black-box, white-box, static, and dynamic testing—the raw skills of a software tester. Web pages are the perfect means to practice what you've learned. You don't have to go out and buy different programs—you can simply jump to a web page, one of your favorites or a completely new one, and begin testing.
The easiest place to start is by treating the web page or the entire website as a black box. You don't know anything about how it works, you don't have a specification, you just have the website in front of you to test. What do you look for?
Figure 14.2 shows a screen image of Apple's website, www.apple.com, a fairly straightforward and typical website. It has all the basic elements—text, graphics, hyperlinks to other pages on the site, and hyperlinks to other websites. A few of the pages have form fields in which users can enter information and a few pages play videos. One interesting thing about this site that's not so common is that it's localized for 27 different locales, from Asia to the UK.
If you have access to the Internet, take some time now and explore Apple's website. Think about how you would approach testing it. What would you test? What would your equivalence partitions be? What would you choose not to test?
After exploring a bit, what did you decide? Hopefully you realized that it's a pretty big job. If you looked at the site map (www.apple.com/find/sitemap.html), you found links to more than 100 different sub-sites, each one with several pages.
NOTE
When testing a website, you first should create a state table (see Chapter 5, “Testing the Software with Blinders On”), treating each page as a different state with the hyperlinks as the lines connecting them. A completed state map will give you a better view of the overall task.
Thankfully, most of the pages are fairly simple, made up of just text, graphics, links, and the occasional form. Testing them isn't difficult. The following sections give some ideas of what to look for.
Web page text should be treated just like documentation and tested as described in Chapter 12, “Testing the Documentation.” Check the audience level, the terminology, the content and subject matter, the accuracy—especially of information that can become outdated—and always, always check spelling.
NOTE
Don't rely on spell checkers to be perfect, especially when they're used on web page content. They might only check the regular text but not what's contained in the graphics, scrolling marquees, forms, and so on. You could perform what you think is a complete spell check and still have misspellings on the page.
If there is contact information such as email addresses, phone numbers, or postal addresses, check them to make sure that they're correct. Make sure that the copyright notices are correct and dated appropriately. Test that each page has a correct title. This text appears in the browser's title bar (upper-left corner of Figure 14.2) and what is listed by default when you add the page to your favorites or bookmarks.
An often overlooked type of text is called ALT text, for ALTernate text. Figure 14.3 shows an example of ALT text. When a user puts the mouse cursor over a graphic on the page he gets a pop-up description of what the graphic represents. Web browsers that don't display graphics use ALT text. Also, with ALT text blind users can use graphically rich websites—an audible reader interprets the ALT text and reads it out through the computer's speakers.
NOTE
Not all browsers support display of the ALT text. Some only show simple TITLE text in the tooltip—or nothing at all. Since this could prevent the blind from using the Web, it should be considered a serious accessibility bug.
Check for text layout issues by resizing your browser window to be very small or very large. This will reveal bugs where the designer or programmer assumed a fixed page width or height. It will also reveal hard-coded formatting such as line breaks that might look good with certain layouts but not with others.
Links can be tied to text or graphics. Each link should be checked to make sure that it jumps to the correct destination and opens in the correct window. If you don't have a specification for the website, you'll need to test whether the jump worked correctly.
Make sure that hyperlinks are obvious. Text links are usually underlined, and the mouse pointer should change (usually to a hand pointer) when it's over any kind of hyperlink—text or graphic.
If the link opens up an email message, fill out the message, send it, and make sure you get a response.
Look for orphan pages, which are part of the website but can't be accessed through a hyperlink because someone forgot to hook them up. You'll likely need to get a list from the web page's designer of the expected pages and compare that with your own state table.
Many possible bugs with graphics are covered later under usability testing, but you can check a few obvious things with a simple black-box approach. For example, do all graphics load and display properly? If a graphic is missing or is incorrectly named, it won't load and the web page will display an error where the graphic was to be placed (see Figure 14.4).
If text and graphics are intermixed on the page, make sure that the text wraps properly around the graphics. Try resizing the browser's window to see if strange wrapping occurs around the graphic.
How's the performance of loading the page? Are there so many graphics on the page, resulting in a large amount of data to be transferred and displayed, that the website's performance is too slow? What happens if you test on a slow dialup connection instead of your company's fast local area network?
Forms are the text boxes, list boxes, and other fields for entering or selecting information on a web page. Figure 14.5 shows a simple example from Apple's website. It's a signup form for potential Mac developers. There are fields for entering your first name, middle initial, last name, and email address. There's an obvious bug on this page—hopefully it's fixed by the time you read this.
Figure 14.5. Make sure your website's form fields are positioned properly. Notice in this Apple Developer signup form that the middle initial (M.I.) field is misplaced.
Test forms just as you would if they were fields in a regular software program—remember Chapter 5? Are the fields the correct size? Do they accept the correct data and reject the wrong data? Is there proper confirmation when you finally press Enter? Are optional fields truly optional and the required ones truly required? What happens if you enter 9999999999999999999999999999?
Your website may contain features such as a hit counter, scrolling marquee text, changing advertisements, or internal site searches (not to be confused with search engines that search the entire Web). When planning your tests for a website, take care to identify all the features present on each page. Treat each unique feature as you would a feature in a regular program and test it individually with the standard testing techniques that you've learned. Does it have its own states? Does it handle data? Could it have ranges or boundaries? What test cases apply and how should they be equivalence classed? A web page is just like any other software.
You're already familiar with black-box and white-box testing, but another type of testing, gray-box testing, is a mixture of the two—hence the name. With gray-box testing, you straddle the line between black-box and white-box testing. You still test the software as a black-box, but you supplement the work by taking a peek (not a full look, as in white-box testing) at what makes the software work.
Web pages lend themselves nicely to gray-box testing. Most web pages are built with HTML (Hypertext Markup Language). Listing 14.1 shows a few lines of the HTML used to create the web page shown in Figure 14.6.
Example 14.1. Sample HTML Showing Some of What's Behind a Web Page
<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <meta name="GENERATOR" content="Microsoft FrontPage 4.0"> <title>Superior Packing Systems</title> <meta name="Microsoft Theme" content="sandston 111, default"> <meta name="Microsoft Border" content="t, default"> </head> <body background="_themes/sandston/stonbk.jpg" bgcolor="#FFFFCC" text="#333333" link="#993300" vlink="#666633" alink="#CC6633"> <!--msnavigation--><table border="0" cellpadding="0" cellspacing="0" width="100%"><tr><td><!--mstheme--><font face="Arial, Helvetica"> <h1 align="center"><!--mstheme--><font color="#660000"> <img src="_derived/index.htm_cmp_sandston110_bnr.gif" width="600" height="60" border="0" alt="Superior Packing Systems"><br> <br> <a href="./"><img src="_derived/home_cmp_sandston110_gbtn.gif" width="95" height="20" border="0" alt="Home" align="middle"></a> <a href="services.htm"> <img src="_derived/services.htm_cmp_sandston110_gbtn.gif" width="95" height="20" border="0" alt="Services" align="middle"></a> <a href="contact.htm"><img src="_derived/contact.htm_cmp_sandston110_gbtn.gif" width="95" height="20" border="0" alt="Contact Us" align="middle"> </a><!--mstheme--></font></h1>
NOTE
If you're not familiar with creating your own website, you might want to read a little on the subject. An introductory book such as Sams Teach Yourself to Create Web Pages in 24 Hours, would be a great way to learn the basics and help you discover a few ways to apply gray-box testing techniques.
HTML and web pages can be simply tested as a gray box because HTML isn't a programming language that's been compiled and inaccessible to the tester—it's a markup language. In the early days of word processors, you couldn't just select text and make it bold or italic. You had to embed markups, sometimes called field tags, in the text. For example, to create the bolded phrase
This is bold text.
you would enter something such as this into your word processor:
[begin bold]This is bold text.[end bold]
HTML works the same way. To create the line in HTML you would enter
<b>This is bold text.</b>
HTML has evolved to where it now has hundreds of different field tags and options, as evidenced by the HTML in Listing 14.1. But, in the end, HTML is nothing but a fancy old-word-processor-like markup language. The difference between HTML and a program is that HTML doesn't execute or run, it just determines how text and graphics appear onscreen.
TIP
To see the HTML for a web page, right-click a blank area of the page (not on a graphic) and select View Source (IE) or View Page Source (Firefox) from the menu.
Since HTML isn't a programming language and is so easy for you, as the tester, to view, you might as well take advantage of it and supplement your testing. If you're a black-box tester, it's the perfect opportunity to start moving toward white-box testing.
Start by learning to create your own simple web pages. Learn the basic and common HTML tags. Look at the HTML for many different pages on the Web, see what techniques are used and how those techniques make things work on the page. Once you become familiar with HTML, you'll be able to look at the web pages you're testing in a whole new way and be a more effective tester.
In Figure 14.1, you saw an example of a web page with much static content in the form of text and images. This static content was most likely created with straight HTML. That same web page also has customizable and dynamic changing content. Remember, HTML isn't a programming language—it's merely a tagging system for text and graphics. To create these extra dynamic features requires the HTML to be supplemented with programming code that can execute and follow decision paths.
You've likely heard of the popular web programming languages and technologies that can be used to create these types of features: DHTML, Java, JavaScript, ActiveX, VBScript, Perl, CGI, ASP, and XML. As explained in Chapters 6, “Examining the Code,” and 7, “Testing the Software with X-Ray Glasses,” to apply white-box testing, you don't necessarily need to become an expert in these languages, just familiar enough to be able to read and understand them and to devise test cases based on what you see in the code.
This chapter can't possibly go into all the details of white-box testing a website, but several features could be more effectively tested with a white-box approach. Of course, they could also be tested as a black-box, but the potential complexity is such that to really make sure you find the important bugs that you have some knowledge of the website's system structure and programming:
Dynamic Content. Dynamic content is graphics and text that changes based on certain conditions—for example, the time of day, the user's preferences, or specific user actions. It's possible that the programming for the content is done in a simple scripting language such as JavaScript and is embedded within the HTML. This is known as client-side programming. If it is, you can apply gray-box testing techniques when you examine the script and view the HTML. For efficiency, most dynamic content programming is located on the website's server; it's called server-side programming and would require you to have access to the web server to view the code.
Database-Driven Web Pages. Many e-commerce web pages that show catalogs or inventories are database driven. The HTML provides a simple layout for the web content and then pictures, text descriptions, pricing information, and so on are pulled from a database on the website's server and plugged into the pages.
Programmatically Created Web Pages. Many web pages, especially ones with dynamic content, are programmatically generated—that is, the HTML and possibly even the programming is created by software. A web page designer may type entries in a database and drag and drop elements in a layout program, press a button, and out comes the HTML that displays a web page. If this sounds scary, it's really no different than a computer language compiler creating machine code. If you're testing such a system, you have to check that the HTML it creates is what the designer expects.
Server Performance and Loading. Popular websites might receive millions of individual hits a day. Each one requires a download of data from the website's server to the browser's computer. If you wanted to test a system for performance and loading, you'd have to find a way to simulate the millions of connections and downloads. Chapter 15, “Automated Testing and Test Tools,” introduces the techniques and tools you can use to do this.
Security. As you learned in the previous chapter, website security issues are always in the news as hackers try new and different ways to gain access to a website's internal data. Financial, medical, and other websites that contain personal data are especially at risk and require intimate knowledge of server technology to test them for proper security.
It's time to get back to what you can do, today, to test a web page. Recall from Chapters 8, “Configuration Testing,” and 9, “Compatibility Testing,” what configuration and compatibility testing are. Configuration testing is the process of checking the operation of your software with various types of hardware and software platforms and their different settings. Compatibility testing is checking your software's operation with other software. Web pages are perfect examples of where you can apply this type of testing.
Assume that you have a website to test. You need to think about what the possible hardware and software configurations might be that could affect the operation or appearance of the site. Here's a list to consider:
Hardware Platform. Is it a Mac, PC, PDA, MSNTV, or a WiFi wristwatch? Each hardware device has its own operating system, screen layout, communications software, and so on. Each can affect how the website appears onscreen.
Browser Software and Version. There are many different web browsers and browser versions. Some run on only one type of hardware platform, others run on multiple platforms. Some examples are AOL 9.0, Firefox 1.0, Internet Explorer 5.0 and 6.0, Pocket IE, Netscape 7.2, and Opera 7.54.
Each browser and version supports a different set of features. A website may look great under one browser and not display at all under another. Web designers can choose to design a site using the least common denominator of features so that it looks the same on all of them, or write specialized code to make the site work best on each one. They may even choose to only support a few of the most popular browsers. How would this impact your testing?
Browser Plug-Ins. Many browsers can accept plug-ins or extensions to gain additional functionality. An example of this would be to play specific types of audio or video files.
Browser Options. Most web browsers allow for a great deal of customization. Figure 14.7 shows an example of this. You can select security options, choose how ALT text is handled, decide what plug-ins to enable, and so on. Each option has potential impact on how your website operates—and, hence, is a test scenario to consider.
Video Resolution and Color Depth. Many platforms can display in various screen resolutions and colors. A PC running Windows, for example, can have screen dimensions of 640×480, 800×600, 1,024×768, 1280×1024, and up. Mobile devices have tiny screens with very different resolutions. Your website may look different, or even wrong, in one resolution, but not in another. Text and graphics can wrap differently, be cut off, or not appear at all.
The number of colors that the platform supports can also impact the look of your site. PCs typically support as few as 256 colors and as many as 232. Could your website be used on a mobile system with only 16 colors?
Text Size. Did you know that a user can change the size of the text used in the browser? Could your site be used with very small or very large text? What if it was being run on a small screen, in a low resolution, with large text?
Modem Speeds. Enough can't be said about performance. Someday everyone will have high-speed connections with website data delivered as fast as you can view it. Until then, you need to test that your website works well at a wide range of modem speeds.
If you consider all the possibilities outlined here, testing even the simplest website can become a huge task. It's not enough that the website looks good on your PC—if you want to ensure that it works well for its intended audience, you need to research the possible configurations they might have. With that information, you can create equivalence partitions of the configurations you feel are most important to test.
A good place to start your research is www.websidestory.com and www.upsdell.com/BrowserNews/stat.htm. These sites have frequently updated surveys related to technology, browsers, video resolutions, etc. These are a great first step in deciding what configurations are popular and which direction they are trending.
Usability and websites are sometimes mutually exclusive terms. You've no doubt seen pages that are difficult to navigate, outdated, slow, or just plain ugly. Not surprisingly, these sites were probably never seen by a software tester. A programmer or someone with little or no design experience (or maybe too much design experience) created the pages and uploaded them for the world to see without considering how usable they were.
As described in Chapter 11, “Usability Testing,” usability testing is a difficult process to define. What looks bad to you might look great to someone else—some people think that Elvis on velvet is art. Fortunately, following and testing a few basic rules can help make websites more usable.
Jakob Nielsen, www.useit.com, a respected expert on website design and usability, has performed extensive research on website usability. The following list is adapted from his Top Ten Mistakes in Web Design:
Gratuitous Use of Bleeding-Edge Technology. Your website shouldn't try to attract users by bragging about its use of the latest Web technology. It may attract a few nerds, but mainstream users will care more about useful content and the site's ability to offer good customer service. Using the latest and greatest technology before it's even released is a sure way to discourage users; if their system crashes while visiting your site, you can bet that many of them won't be back. Unless you're in the business of selling Internet products or services, it's better to wait until some experience has been gained with the technology. When desktop publishing was young, people put 20 different fonts in their documents; try to avoid similar design bloat on the Web.
Scrolling Text, Marquees, and Constantly Running Animations. Never allow page elements that move incessantly. Moving images have an overpowering effect on human peripheral vision. A web page shouldn't emulate Times Square in New York City in its constant attack on the human senses—give your user some peace and quiet to actually read the text!
Long Scrolling Pages. Users typically don't like to scroll beyond the information visible onscreen when a page comes up. All critical content and navigation options should be on the top part of the page. Recent studies have shown that users are becoming more willing to scroll now than they were in the early years of the Web, but it's still a good idea to minimize scrolling on navigation pages.
Non-Standard Link Colors. Hyperlinks to pages that users haven't seen should be blue; links to previously seen pages should be purple or red. Don't mess with these colors because the ability to understand which links have been followed is one of the few navigational aids that's standard in most web browsers. Consistency is key to teaching users what the link colors mean.
Outdated Information. Your development team should have a web “gardener”—someone to root out the weeds and replant the flowers as the website changes. Unfortunately, most teams would rather spend their time creating new content than doing maintenance. In practice, maintenance is a cheap way of enhancing the content on your website since many old pages keep their relevance and should be linked into the new pages. Of course, some pages are better off being removed completely from the server after their expiration date.
Overly Long Download Times. Traditional human-factor guidelines indicate that 0.1 second is about the limit for users to feel that the system is reacting instantaneously. One second is about the limit for a user's flow of thought to stay uninterrupted. Ten seconds is the maximum response time before a user loses interest.
On the Web, users have been trained to endure so much suffering that it may be acceptable to increase this limit to 15 seconds for a few pages. But don't aim for this—aim for less.
Lack of Navigation Support. Don't assume that users know as much about your site as you do. They will always have difficulty finding information, so they need support in the form of a strong sense of structure and place. Your site's design should start with a good understanding of the structure of the information space and communicate that structure explicitly to users. Provide a site map to let users know where they are and where they can go. The site should also have a good search feature because even the best navigation support will never be enough.
Orphan Pages. Make sure that all pages include a clear indication of what website they belong to since users may access pages directly without coming in through your home page. For the same reason, every page should have a link to your home page as well as some indication of where they fit within the structure of your information space.
Complex Website Addresses (URLs). Even though machine-level addressing like the URL should never have been exposed in the user interface, it's there and research has found that users actually try to decode the URLs of pages to infer the structure of websites. Users do this because of the lack of support for navigation (see above) and sense of location in current web browsers. Thus, a URL should contain human-readable names that reflect the nature of the website's contents.
Also, because users often type in a URL, the website author should try to minimize the risk of typos by using short names with all lowercase characters and no special characters (many people don't know how to type a ~).
Using Frames. Frames are an HTML technology that allows a web page to display another web page within itself, hence the name frame—like a picture frame. Splitting a page into frames can confuse users since frames break the fundamental user model of the web page. All of a sudden they can't bookmark the current page and return to it (the bookmark points to another version of the frameset), URLs stop working, and printouts become difficult. Even worse, the predictability of user actions goes out the door—who knows what information will appear where and when they click a link? Many website authors, to avoid problems, forego the use of frames and simply open another window.
If you're testing a website, take advantage of your tester's license to report bugs on usability. Read up on basic user interface design techniques and learn what makes for good usability. A good source of information is a Microsoft research document titled, “Improving Web Site Usability and Appeal.” Its web address is msdn.microsoft.com/workshop/management/planning/improvingsiteusa.asp. This document provides a list of best practices that Microsoft discovered while designing content for its MSN websites. Don't be put off by the date, 1997, on the document. Good design is timeless.
The last part of this chapter is somewhat a lead-in to the next chapter of the book, “Automated Testing and Test Tools.”
You may have wondered as you read this chapter how you could possibly have enough time to thoroughly test a large and complex website. The simple act of clicking all the hyperlinks to make sure that they're valid could take a great deal of time. Add in testing the basic functionality of the website's features, doing configuration and compatibility testing, and figuring out a way to test performance and loading by simulating thousands or even millions of users, and you have a big job.
Thankfully, you don't have to do all this testing manually. Testing tools are available, both free and for purchase, that will make your job much easier. Several tools are located at www.netmechanic.com. This site, and others like it, provides easy-to-use tools that will automatically examine your website and test it for things such as browser compatibility, performance problems, broken hyperlinks, HTML standard adherence, and spelling. They can even tell you what graphics on your site are possibly too large and are causing it to be slow. Such tools can save you many hours of what would normally be manual work. Look at them for an idea of what you'll learn about in Chapter 15.
This chapter wraps up Part III which covered a lot of ground—from video card settings to Hungarian localization to ugly websites. These topics are just a small piece of the total software world. That diversity is what makes software testing a limitless challenge. Every day new and exciting types of software are released, always pushing the technology forward and creating unique and interesting testing problems to solve. Website testing is a good topic for this chapter, today, but who knows what it will be in the future.
Hopefully in reading the chapters in Part III you realized that you or a small test team could easily be overwhelmed by the magnitude of testing even a small software product or website. A few hundred lines of code written by a single programmer with all its possible platforms, configurations, languages, and users could require dozens or even hundreds of testers to thoroughly test them. The combinations and permutations are endless and, even with careful equivalence partitioning to reduce the number of cases, the job can still seem too big to handle.
In the next two chapters, you'll learn how to leverage both tools and people to bring this huge task down to a manageable size.
These quiz questions are provided for your further understanding. See Appendix A, “Answers to Quiz Questions,” for the answers—but don't peek!