Notes
Chapter 2
1. This reference is the first of many to the RISKS section of the ACM SIGSOFT Software Engineering Notes, cited by volume and issue number, and, in the more recent issues, page number. The month and year are omitted, and can be derived from the table in Appendix A; in this case, “SEN 12, 1” corresponds to January 1987.
2. Telephony, p. 11, January 22, 1990.
3. The New York Times, National Edition, October 16, 1990, page A14; SEN 16, 1.
4. See an Associated Press item, January 5, 1991; SEN, 16, 1, 14.
5. See an article by Carl Hall, San Francisco Chronicle, July 16, 1991.
6. See John Noble Wilford, The New York Times, August 26, 1985.
7. SEN 11, 5; San Francisco Chronicle, August 6, 1986.
8. Up to and including STS-27, the STS mission-planning serial number also indicates the sequence number of the launch. Beginning with STS-28, which was actually the thirtieth mission, and STS-29, which was the twenty-eighth mission, the numbers ceased to reflect the chronological order of the launch. The STS numbers are given here to provide specificity.
9. This summary of “Wrong Computer Instructions Were Given to Discovery Before Liftoff” by AP science writer Lee Siegel, October 10, 1990, was contributed by Fernando Pereira; SEN 16, 1.
10. See Aviation Week and Space Technology, May 25, 1992, p. 79, and June 8, 1992, p. 69; SEN 17, 3.
11. SEN 18, 3, A-14, summarizing San Francisco Chronicle, April 6, 1993, p. A2, and April 7, 1993, p. A3. See also a detailed subsequent report by Paul Robichaux that appeared in the on-line RISKS, 14, 47, April 7, 1993.
12. San Francisco Chronicle, October 15, 1993, p. A13.
13. From The Compleat Computer, which reprinted it with permission from Fortune, March 1967—“Help Wanted: 50,000 Programmers”—by Gene Bylinsky, Fortune; contributed to RISKS by Ed Joyce. See SEN 10, 5.
14. A detailed account of this case is given by Paul Ceruzzi in Beyond the Limits—Flight Enters the Computer Age, Smithsonian Air and Space Museum (Appendix), 1989, quoted in SEN 14, 5.
15. From San Francisco Chronicle wire services, Aug 28, 1986; see SEN 11, 5.
16. See Joseph M. Fox, Software and Its Development, Prentice-Hall, Englewood Cliffs, New Jersey, pp. 187-189; contributed by David B. Benson; see also SEN 9, 1.
17. The New York Times, August 3, 1993.
18. See an Associated Press item, San Francisco Chronicle, July 20, 1993, p. 3.
19. See Science, September 16, 1988; San Francisco Chronicle, September 10, 1988, p. All; SEN 13, 4; 14, 2, and Science 245, p. 1045, September 8, 1989; SEN 14, 6 for Phobos 1 and 2, respectively.
20. See an article by Kathy Sawyer, “NASA Change in Flight Plan Blamed for Mars Probe Mishap,” The Washington Post, January 19, 1994.
21. San Francisco Chronicle, October 9, 1993.
22. This report is based on articles by David Perlman in the San Francisco Chronicle, August 25-26, 1993; by John Noble Wilford of The New York Times, August 23; and from the Los Angeles Times, August 24.
23. Newsweek, April 24, 1994.
24. See various press reports in The New York Times, April 15 and 18, and July 1, 1994.
25. See The Washington Post, April 15, 1994.
26. See an article by Christopher Bellamy in The Independent, April 23, 1992, and a later item in The Guardian, October 24, 1992.
27. See an article by Steve Komarow in USA Today, October 27, 1993, p. 1A, which also noted the sinking of the USS Thresher in 1963, killing 129 men aboard, apparently because of a leak. Both submarines were nuclear-powered.
28. Daniel F. Ford, The Button: The Pentagon’s Strategic Command and Control System, Simon and Schuster, New York, 1985.
29. For example, see E.C. Berkeley, The Computer Revolution, Doubleday, New York, pp. 175-177, 1962.
30. From a talk by Mark Groves, reported by Earl Boebert in SEN 5, 2.
31. Aviation Week and Space Technology, pp. 28-30, June 10, 1991; SEN 16, 3.
32. Flight International, October 13-19, 1993.
33. The Washington Post, August 21, 1987; San Francisco Chronicle, p. A5, May 11, 1988; SEN 12, 4; 13, 1; 13, 3; 14, 5; 15, 5.
34. Flight International, April 1, 1989; SEN 14, 2; 14, 5; 15, 3; 16, 3.
35. There were various reports on-line and in SEN 13, 4; 14, 2; 14, 5; 15, 3; 16, 3. However, the analysis of this case has been marred by considerable confusion.
36. Flight International, October 13-19, 1993.
37. Flight International, September 15-21, 1993.
38. See the New York Review, April 25, 1985; SEN 9, 1; 10, 3; 12, 1; and an article by Murray Sayle, “Closing the File on Flight 007,” The New Yorker, December 13, 1993, pp. 90-101.
39. Flight International, February 27 and March 5, 1991; SEN 16, 2.
40. See a letter to Aviation Week and Space Technology, January 30, 1989; SEN 14, 2.
41. IEEE Spectrum, August 1991, p. 58; SEN 16, 4; 17, 1; 18, 1, 24.
42. San Francisco Chronicle, January 15, 1987.
43. Literally, Thus passes away the glory of the world; alternatively, ailing transit systems can create a glorious start of the week—for the news media, perhaps, but certainly not for commuters.
44. Reported in SEN 14, 6 by Michael Trout, excerpting an article in Call Board of the Mohawk & Hudson Chapter National Railway Historical Society.
45. San Francisco Chronicle, February 9, 1986; SEN 11, 2.
46. John E. Woodruff, Baltimore Sun, datelined Tokyo, in San Francisco Chronicle, May 15, 1991, p.A7; SEN 16, 3.
47. “Results of Train Accident Investigations Indicate Driver’s Neglect of Traffic Signals Direct Cause of Accident” — abridged from the China Times, November 23, 1991, by Jymmi C. Tseng in SEN 17, 1.
48. Debora Weber-Wulff, Technische Fachhochschule Berlin, based on information from Tagespiegel, April 14, 1993; SEN 18, 3, A-3.
49. See “British Train Accidents Signal Systemic Problems” by Fred Guterl and Erin E. Murphy, IEEE Institute, May 1989, p. 4; excerpted in SEN 14, 6, summarized by Jon Jacky.
50. Patrick Donovan, BR Signalmen “‘Worked Blind’; Computer Software Problems Admitted at Key Commuter Train Centre,” The Guardian, July 23, 1990, front page; excerpted by Pete Mellor, in SEN 15, 5.
51. Noted in SEN 16, 2 by Peter Mellor from an article by Dick Murray, (London) Evening Standard, January 24, 1991.
52. Roy Hodson, Financial Times, March 15, 1990, excerpted by Brian Randell in SEN 15, 3.
53. Stephen Page, summarizing an article by Dick Murray, (London) Evening Standard, April 12, 1990, with further comment from Gavin Oddy; see SEN 15, 3.
54. Tony Collins, “Autumn Leaves Fox BR’s Signal System,” Computer Weekly, November 7, 1991, noted by Graeme Tozer, with further comments from Geraint Jones, in SEN 17, 1.
55. The Denver Post, November 12, 1991, noted by Bob Devine in SEN 17, 1.
56. Noted by Olivier M.J. Crepin-Leblond in SEN 17, 1.
57. San Francisco Chronicle, July 8, 1987; noted in SEN 12, 3.
58. Harry W. Demoro and Carl Nolte, San Francisco Chronicle, December 10, 1986, p. 2; see SEN 12, 1.
59. Carl Nolte, San Francisco Chronicle, April 6, 1993; Phillip Matier and Andrew Ross, San Francisco Chronicle, April 7, 1993, summarized in SEN 18, 3, A-3.
60. Computing Australia, August 29, 1988; George Michaelson in SEN 13, 4.
61. “Worlds of Fun Timber Wolf Incident Blamed on Computer,” ACE News (from The American Coaster Enthusiasts), Volume XII, Issue 6, May 1990, contributed by Gary Wright, in SEN 15, 3.
62. See an article by Chuck Ayers, The Morning Call, Allentown, Pennsylvania, July 28, 1993, p. B5, contributed to SEN 18, 4 by Steven D. Walter.
63. Deeann Glamser, USA Today, February 25, 1987; SEN 12, 2; see also subsequent items in SEN 14, 2 and SEN 15, 2.
64. This report is based on comments by John Sullivan, The New York Times, August 12, 1992, and subsequent Vineyard Gazette reports.
65. The Independent, June 5, 1992, from Brian Randell in SEN 17, 4.
66. Penelope M. Carrington, “Ship Makes List—the Hard Way,” The Seattle Times, August 5, 1992, p. D1, from Jon Jacky in SEN 17, 4.
67. See The New York Times, August 14 and 24, 1985; SEN 10, 5, 6.
68. I. Nimmo, S.R. Nunns, and B.W. Eddershaw, “Lessons Learned from the Failure of a Computer System Controlling a Nylon Polymer Plant,” Safety and Reliability Society Symposium, Altrincham, United Kingdom, November 1987.
69. These are just two of a series of incidents noted by Jon Jacky, excerpted from an article by Trevor Kletz, “Wise After the Event,” Control and Instrumentation (UK), 20, 10, October 1988, pp. 57, 59, in SEN 14, 2.
70. Based on a report contributed by Meine van der Meulen, in SEN 18, 2, 7, which in turn was based on Piet van Beek, “FACTS, Database for Industrial Safety Acc. 11057, Extended abstract,” The Netherlands Organization for Applied Scientific Research, Department of Industrial Safety, Apeldoorn, The Netherlands, July 8, 1992.
71. Associated Press, Johannesburg, South Africa, December 28, 1988, from Nicholas Gegg in SEN 14, 2.
72. The Ottawa Citizen, June 28, 1989, p. B1, and June 29, 1989, p. A1-A2, noted by Walter Roberson in SEN 14, 5.
73. Montreal Gazette, September 9, 1988, noted by Henry Cox in SEN 13, 4.
74. Toronto Star, April 20, 1989, noted by Mark Brader in SEN 14, 5.
75. San Jose Mercury, November 16, 1988, noted by Ira Greenberg in SEN 14, 1.
76. Dick Lilly, Seattle Times, January 8, 1990, p. 1, reported by Jon Jacky in SEN 15, 2.
77. Asbury Park Press, August 30, 1992, from George Sicherman in SEN 18, 1, 8.
78. This case was reported by a former factory employee, Joe Pistritto, in the early days of the on-line Software Engineering Digest.
79. Electronic Engineering Times, December 21, 1981, noted by Thomas Litant in SEN 10, 3.
80. See SEN 10, 2 and “In the Lion’s Cage,” Forbes, October 7, 1985, contributed by Bill Keefe in SEN 11, 1.
81. Abstracted in SEN 16, 1 by Brad Dolan from the Maryville/Alcoa (Tenn.) Daily Times, September 10, 1990, p. 1; Brad commented that “This looks like (1) poor work practice and (2) poor vehicle design.”
82. Ron Cain, contributed to SEN 11, 3 by Bill Park.
83. From a Reuters item in the June 26, 1985, issue of the Halifax Gazette, via Digital Review, September 15, 1986, in SEN 11, 5 courtesy of Brad Dolan.
84. See a UPI item, October 10, 1992, noted by Les Earnest in SEN 18, 1, 7, and a subsequent San Francisco Chronicle article by Mary Madison, December 13, 1993, p. A19.
85. Peter Boyer, “At Networks, Cheap Is Chic, so Please Pardon the Robots,” The New York Times, Television, May 2, 1988, contributed to SEN 13, 3 by Donn Seeley.
86. See Motor Trend, November 1986, in SEN 13, 3, noted by Tom Slone.
87. San Francisco Chronicle, January 3, 1993, p. B-6.
88. Science, 262, October 22 1993, p. 495.
89. San Francisco Chronicle, August 27, 1993, p. A8.
90. Joan Temple Dennett, Science 80, 1980; SEN 5, 1.
91. For background on pacemakers, see PACE, vol. 7, Nov/Dec 1984, part 2.
92. See the series of three articles in The New Yorker by Paul Brodeur, June 12, 19, and 26, 1989; see also SEN 11, 3; 11, 5; 15, 5, including discussion of an article on VDT radiation by Paul Brodeur in MacWorld.
93. Reported by Edward Joyce from a United Press International item, March 19, 1983; SEN 10, 3.
94. Computer Weekly, February 20, 1992; SEN 17, 3.
95. From an article by Christine Spolar of The Washington Post, in The Times-Picayune, New Orleans, February 2, 1992, p. A-22; submitted by Sevilla Finley.
96. Journal of the American Medical Association, December 8, 1989, page 3132; Chemical and Engineering News, p. 168, February 26, 1989; SEN 15, 2.
97. See New Scientist, November 14, 1992; see also SEN 17, 3; SEN 18, 1, 26-28.
98. The primary early public source on the Therac-25 accidents is the series by Edward Joyce in the weekly American Medical News, beginning on October 3, 1986. Another early report on the Therac-25 is given by Jon Jacky in Inside Risks, Communications of the ACM, 33, 12, 138, December 1990. He notes several background sources. For background on the medical electronics industry and accounts of computer-related product recalls, see H. Bassen, J. Silberberg, F. Houston, W. Knight, C. Christman, and M. Greberman, “Computerized Medical Devices: Usage Trends, Problems, and Safety Technology,” in Proc. 7th Annual Conference of IEEE Engineering in Medicine and Biology Society, September 27-30, 1985, Chicago, Illinois, pages 180-185. See also “Product Recalls,” Mylene Moreno, The Boston Globe, April 10, 1989, p. 14, cited by B.J. Herbison. Food and Drug Administration Director Frank Young described his agency’s views in Annals of Internal Medicine 106, 4, April 1987, 628-629.
99. These cases are taken mostly from The Washington Post, April 29, 1986.
100. See an extensive and well-written case study by Daniel F. Ford, Three Mile Island: Thirty Minutes to Meltdown, Viking Press, 1981, Penguin paperback, 1982, noted by J. Paul Holbrook in SEN 11, 3.
101. The Washington Post, May 24, 1986, in SEN 11, 3.
102. See The Washington Post, May 24, 1986.
103. David E. Kaplan (Center for Investigative Reporting in San Francisco), “It’s Not Russia’s First Meltdown,” San Francisco Chronicle, May 1, 1986.
104. San Francisco Examiner, April 14, 1991, p. A-6; San Francisco Chronicle, April 14, 1991, p. A10, and April 17, 1991.
105. The Washington Post, May 22, 1986.
106. See The New York Times, August 16, 1986.
107. The New York Times, July 25, 1986.
108. The New York Times, August 18, 1986.
109. San Francisco Chronicle, July 24, 1991, p. A8, in SEN 16, 4.
110. San Francisco Chronicle, June 4, 1986.
111. These two paragraphs are adapted from an editorial by the author in SEN 4, 2.
112. Eliot Marshall, “NRC Takes a Second Look at Reactor Design,” Science, 207, pp. 1445-48, March 28, 1980, contributed to SEN 10, 3 by Nancy Leveson.
113. Susan Watts, “Computer Watch on Nuclear Plant Raises Safety Fears,” The (London) Independent, October 13, 1991, contributed by Antony Upward. Upward also submitted two long articles that appeared in the on-line forum, RISKS 12, 49. A precursor of this item on Sizewell was included in SEN 16, 2, April 1991.
114. Tom Wilkie and Susan Watts, “Sellafield Safety Computer Fails,” The Independent, November 24, 1991, noted by Peter Ilieve. See SEN 17, 1 and SEN 18, 1, 27.
115. “Group Questions Software’s Reliability after Bruce Accident,” Canadian Press, February 1, 1990, noted by Mark Bartelt; SEN 15, 2 and SEN 16, 2.
116. Abstracted from Nucleonics Week 32, 1, January 3, 1991, and 2, January 10, 1991, McGraw-Hill, contributed to SEN 15, 1 by Martyn Thomas.
117. New Scientist, February 17, 1990, p. 18, noted in SEN 15, 2 by Steve Strassmann.
118. “Sabotage Fails—Virus in Power-Plant Program for the Lithuanian Atomic-Power Plant in Ignalina Vaccinated,” Berliner Zeitung ([East] Berlin), February 3, 1992, translated and contributed to SEN 17, 2 by Debora Weber-Wulff, Institut für Informatik, Berlin.
119. See Keith Schneider, “Power Surge Causes Failure of Systems in New York Nuclear Plant,” The New York Times, August 15, 1991, noted in SEN 16, 4.
120. Condensed from the Albany (NY) Times Union, April 26, 1989, p. B-17, noted by William Randolph Franklin in SEN 14, 5.
121. San Francisco Chronicle, October 4, 1984, SEN 9, 5.
122. San Jose Mercury News, January 19, 1994.
123. This item is excerpted from the Ottawa Citizen, August 8, 1986, noted in SEN 11, 5 by Dan Craigen.
124. Contributed by Bryan MacKinnon, Fermi National Accelerator Lab, in SEN 17, 3.
125. “Lost Squirrel Causes Troublesome Power Surge,” Providence Journal, October 30, 1986, from Ross McKenrick in SEN 12, 1.
126. This event was reported by David L. Edwards in SEN 14, 5.
127. See Science News, May 11, 1989, noted by Peter Scott in SEN 14, 5.
128. This item was noted on the same day by David Lamb in the newsgroup comp.os.linux.announce and also appeared in SEN 19, 1, 4.
129. The time shift seems to have resulted in Braunschweiger Clockwurst. Perhaps in the foreshortening of time, the process controller was related to Ingot Zwergman.
130. New Scientist, September 8, 1988; SEN 13, 4.
131. Roger Fea, “Year Too Long for Money Machines,” New Zealand Herald, January 2, 1993.
132. “Leap-Year Spikes Cashcards,” NZPA, Waikato Times, January 2, 1993.
133. It is perhaps helpful that dates and times prior to the introduction of the Gregorian calendar are not particularly relevant to modern computing.
134. “The Subjectivity of Computers,” Communications of the ACM, 36, 8, 15-18, August 1993, noted in RISKS by Dave Wortman.
Chapter 3
1. The basic notions underlying security vulnerabilities are contained in or referred to in this chapter. Landwehr, Bull, McDermott, and Choi [75] have developed a taxonomy that classifies program security flaws according to the motive (intentional or inadvertent), the time of introduction (during development, maintenance, or operation), and the place of introduction (software or hardware). Subdividing intentional flaws into malicious and nonmalicious types, and further substructuring these types, they provide examples for almost all the classifications. These distinctions are useful for understanding the modalities of misuse, but are often not constructively distinguishable from an operational point of view.
Two books relating more specifically to pest programs and security flaws are edited by Lance Hoffman [57] and by Peter Denning [35]. Each book reprints various relevant papers and contains background on personal-computer viruses — in a few cases with identical material appearing in both. However, recent activities in the personal-computer virus world suggest that someone wishing to stay abreast of the situation must diligently read the VIRUS-L newsgroup. (See Appendix A.)
Bill Cheswick and Steve Bellovin [24] have written an outstanding book on firewalls and Internet security, enumerating many of the major security flaws and recommending what to do about them. This book is an absolute requirement for anyone concerned about the risks of living in the Internet world.
A reader interested in the acts of computer crime, as opposed to the technological modalities, might wish to read books by Donn Parker [119, 120] and Buck BloomBecker [12].
Chapter 4
1. An insightful analysis of such interactions is given by R.I. Cook, “Reflections on a Telephone Cable Severed near Chicago,” SEN 16, 1, 14-16.
2. See The Washington Post, May 24, 1986, and SEN 11, 3.
3. See Science, May 3, 1985, p. 562, and SEN 10, 3.
4. Butler Lampson [72] wrote an early paper recognizing that a hardware memory-protection mechanism cannot provide security unless it is reliable. That concept is readily extended to systems as a whole, in hardware and in software. A hierarchical approach to integrating security and reliability (along with requirements) is suggested in [107].
Chapter 5
1. See SEN 12, 4; 13, 1; 13, 2; 14, 2; 14, 6; and 15, 2. See also RISKS 8, 36 to 38, including a response from “pengo” (an alias for Hans Huebner) in RISKS 8, 37.
2. See SEN 16, 3; 17, 2; see also RISKS 13, 11 and 12.
3. John Markoff, The New York Times, April 4, 1990; SEN 15, 3.
4. An article from the now-defunct Palo Alto Times Tribune, February 7, 1988, is summarized in SEN 13, 2.
5. Los Angeles Times, March 23, 1991; SEN 16, 3.
6. SEN 14, 1; Los Angeles Times, December 23, 1988. See also [12].
7. London Daily Mail, November 2, 1984; SEN 10, 1.
8. Wall Street Journal, August 15, 1985; SEN 10, 5.
9. This situation was reported by a Computer Emergency Response Team (CERT) Internet Security Advisory; see SEN 14, 6.
10. Los Angeles Times, November 5, 1991; SEN 17, 1.
11. The Washington Post, May 31, 1988; Ft. Worth Star-Telegram, September 19, 1988; SEN 13, 3; 13, 4.
12. The Times of London, June 9, 1992; SEN 17, 4.
13. Readers seriously interested in a detailed perspective on viruses should study the VIRUS-L archives, which include pointers to various catalogs such as that of Klaus Brunnstein. There are many books, including Ferbrache [44], Hoffman [57], and Kane [62].
14. See The New York Times, July 30, 1993, p. B5.
15. Ashland, Oregon, news article from August 23, 1986; SEN 11, 5.
16. A few other earlier computer-related perpetrations are reprinted in a special section of the CACM 27, 4, 343-357, April 1984.
17. This message appeared in RISKS 6, 52, April 1, 1988, courtesy of Cliff Stoll, and subsequently in SEN 13, 3, 9-10.
18. SEN 9, 2, 4-5, from an Associated Press report, “Cal Tech Prankster to Get Course Credit for Scoreboard Takeover,” January 3, 1984.
19. Chicago Tribune, February 3, 1993, noted by Tony Scandora in SEN 18, 2, 14.
20. Jay Rolls, Stuttgart, “Computer Cheats Take Cadsoft’s Bait,” from info-mac, in SEN 18, 2, 14, via Gio Wiederhold.
21. We note in Section 4.2 that failures in reliability could alternatively have resulted from malicious human actions; in some other cases, the effects of malicious human actions could alternatively have been caused by hardware malfunctions or software flaws. This duality should be kept in mind here.
22. This computer-communication spy flick was based on Robert Lindsey’s true story of the same name, describing the saga of Andrew Daulton Lee and Christopher John Boyce; Simon and Schuster, 1979.
23. Los Angeles Times, July 24, 1986.
24. See an article by Richard March, PC Week, p. 1, November 27, 1989, SEN 15, 1.
25. SEN 19, 2, 2; The Boston Globe, December 3, 1993.
26. See Datamation, p. 67, December 15, 1986; SEN 12, 1.
27. The Boston Globe, May 16, 1986.
28. Aviation Week, March 11, 1985.
29. Numerous cases of interference problems experienced in commercial aviation are listed in “The Electronic Safety Threat,” Airline Pilot, June 1993, p. 30.
30. See the Martha’s Vineyard Times, p. 1, August 4, 1988, summarized in SEN 13, 4.
31. See Fernando M. Esparza, “The Importance of EMC in the Preparation and Selling of Big Macs,” EMC Technology, September-October 1989; SEN 15, 1.
32. John Darnton, “London Journal,” The New York Times, June 30, 1993, p. A4; SEN, 18, 3, A-10.
33. Noted by Paul Leyland and Mark Huth in SEN 16, 2.
34. See the San Francisco Chronicle, p. A2, February 2, 1989; SEN 14, 2.
35. See Ari L. Goldman, The New York Times, June 30, 1993, p. B6; SEN 18, 3, A-9.
36. The Boston Globe, July 2, 1993, p. 19.
37. With apologies to Thomas Morley, 1557-1603.
38. See John Stebbins, “Fischer Victory Upheld,” St. Petersburg Tribune, April 6, 1993, Florida/Metro section.
39. The Virginia, Durham, Rome, Yonkers, and Michigan cases were discussed in SEN, 15, 1, 10-13. The 1992 cases are documented in SEN, 18, 1, 15-19. For further background, see a report by Roy G. Saltman [143] and a subsequent paper [144]. Saltman also contributed an on-line piece on the role of standards, in SEN, 18, 1, 17. Four papers from the Proceedings of the 16th National Computer Security Conference are relevant, by Saltman [145], Rebecca Mercuri, [92], Peter Neumann [112], and Gary L. Greenhalgh, [55]. See also publications by two nongovernmental organizations, Computer Professionals for Social Responsibility (P.O. Box 717, Palo Alto, CA 94302, including a voluminous collection of study items) and Election Watch (a project spearheaded by Mae Churchill of the Urban Policy Research Institute, 530 Paseo Miramar, Pacific Palisades, CA 90272).
40. Article by Bill Johnson, Los Angeles Herald Examiner, prior to September 14 1987; noted by Bill Weisman in SEN 12, 4.
41. See “Fraudulent Fax Gets Forger Freed” in the San Francisco Chronicle, December 18, 1991, p. A3, excerpted in SEN 17, 1.
42. San Francisco Chronicle, July 6, 1987.
43. See a note from Paul Molenaar, via Patrick van Kleef, in SEN 12, 4.
44. See the San Jose Mercury News, December 14, 1984; SEN 10, 1.
45. From an Associated Press item in the Montreal Gazette, September 22, 1988, via Henry Cox, in SEN 13, 4.
46. SEN 18, 2, 4, from an article by Peter Fimrite in the San Francisco Chronicle, December 30, 1992, p. A14.
47. Ibid.
48. See “New Software Fails to Fix Jail’s Computer System” by Judy Kuhlman, Daily Oklahoman, February 26, 1993, contributed to SEN 18, 2 by Jennifer Smith, who noted that “Having computer-controlled doors with not even a surge protector, not to mention no one in the state running the system, is unfortunately quite typical.” Smith believes that this jail was the same allegedly “escape-proof” jail from which two inmates escaped within the first month of operation.
49. Some of the general issues relating to security are considered in a National Research Council report, Computers at Risk [26], which surveys the situation existing in 1990, and makes various recommendations. A useful book on computer security is provided by Charles Pfleeger [131]. There are few definitive treatises on how to do it right. Morrie Gasser [48] has written an excellent book on designing secure systems. Sape Mullender [97] has edited a useful book on distributed systems, although security is not the driving requirement. System authentication in distributed systems is discussed by Lampson, Wobber, and colleagues [71, 178], based on important earlier work by Needham, Schroeder, and others. An earlier view of distributed-system security is provided by Gasser and others. [49]. Bill Cheswick and Steve Bellovin [24] provide an outstanding analysis of how to attain better security in the Internet. There are many articles and books that inadvertently describe how not to do it, although that can be inferred only from a deeper understanding of the weaknesses of the resulting systems. In addition, system penetrators are examined by Bruce Sterling [161].
Chapter 6
1. Lawrence Ferlinghetti, March 31, 1993, in response to the city of San Francisco, which was trying to impose a $600 license fee for poetry reading.
2. ACM Code of Professional Conduct, Ethical Consideration 5.1.
3. Records, Computers and the Rights of Citizens, Report of the Secretary’s Advisory Committee on Automated Personal Data Systems, U.S. Dept. of Health, Education & Welfare, 1973.
4. R. Rosenberg, “The Role of Computer Scientists in Privacy Policy,” Information Technology Quarterly IX, 2, 24, Summer 1990.
5. “Proposed FBI Crime Computer System Raises Questions on Accuracy, Privacy: Report Warns of Potential Risk Data Bank Poses to Civil Liberties,” The Washington Post, Feb. 13, 1991. The full report, J.J. Horning, P.G. Neumann, D.D. Redell, J. Goldman, and D.R. Gordon, A Review of NCIC 2000, report to the Subcommittee on Civil and Constitutional Rights of the Committee on the Judiciary, United States House of Representatives, February 1989, is available from Computer Professionals for Social Responsibility.
6. L. Winner, “A Victory for Computer Populism,” Technology Review, p. 66, May/June 1991.
7. See “National Crime Information Center: Legislation Needed to Deter Misuse of Criminal Justice Information,” statement of Laurie E. Ekstrand, U.S. General Accounting Office, as testimony before the U.S. House of Representatives Subcommittee on Information, Justice, Agriculture, and Transportation, of the Committee on Government Operations, and the Subcommittee on Civil and Constitutional Rights, of the Committee on the Judiciary, July 28, 1993. The appendix to that testimony documents 61 additional cases of misuses of NCIC and other law-enforcement computer systems. Many of the cited cases are the result of misuse of state and local computer systems rather than the NCIC system. The GAO report recommends that (1) Congress enact legislation with strong criminal sanctions for misuse of NCIC, and (2) NCIC’s security policy requirements be reevaluated.
8. Reported in Harpers, and noted in SEN 17, 3.
9. San Francisco Examiner, November 1, 1992; see also SEN 18, 1, 20.
10. Jay Sharbutt, Los Angeles Times, March 1, 1989; SEN 14, 2.
11. The New York Times, February 7, 1993, p. 32.
12. This item is excerpted from an article by Evan Ramstad, Associated Press, December 19, 1991; see SEN 17, 1.
13. See issues of the on-line Risks Forum, beginning with RISKS 14, 51. Also, see an unclassified description of the process by Dorothy Denning [32].
14. De Volkskrant, February 22, 1991.
15. See a Reuters item in the Los Angeles Times, September 19, 1993.
16. Both of these cases came from articles by Simon Hoggart in the Observer, contributed to SEN 16, 2, 13 by Pete Mellor.
17. The Sydney Morning Herald, July 23, 1988, from John Colville in SEN 13, 4.
18. In The Illuminatus! Trilogy, Robert J. Shea and Robert Anton Wilson ([153], p. 692).
19. See an article by David Burnham, The New York Times, February 12, 1985, with followup items in SEN 11, 1; 12, 4; 13, 2.
20. Mike Zehr in SEN 19, 3, 7, from an article in The Boston Globe, March 7, 1994.
21. Excerpted from an article by Catherine Bowman, “Sly Imposter Robs S.F. Man of Good Name,” San Francisco Chronicle, March 14, 1994, p. 1, contributed by Mike Crawford.
22. Ottawa Citizen, November 23, 1992, from Stanley Chow in SEN 18, 1, 22.
23. See a front-page article by Yasmin Anwar in the San Francisco Chronicle, August 30, 1991, reprinted with permission in the on-line RISKS 12, 20.
24. Chris Hibbert ([email protected]) has written an excellent discussion of SSNs, “What to Do When They Ask for Your Social-Security Number. Social-Security Number FAQ (Frequently Asked Questions).” Updated versions appear periodically in various on-line newsgroups, and can be obtained by anonymous FTP from rtfm.mit.edu in the file /pub/usenet-by-hierarchy/news/answers/ssn-privacy, or by E-mail sent to [email protected] containing the one-line text: send usenet-by-hierarchy/news/answers/ssn-privacy. Sending E-mail with the one-line help will get you general information about the mail server, which also has many other FAQs.
25. Some of these items are from alt.folklore.computers, contributed by John Miller, John Switzer, Jeff Hibbard, Jay Maynard, Joel Sumner, Jeff DelPapa, Hugh J.E. Davies, Terry Kennedy, Jake Richter, Kevin Stevens, Scott Telford, and Brad Heintz; those items were collected for RISKS by Mark Brader. Jay Maynard noted that having a comma before the ‘III’ tends to avoid the problem with his name. The Goleta case was reported by Arthur L. Shapiro.
26. See the on-line RISKS 14, 15-17, December 7-8, 1992, for the entire discussion; see Section 6.5 for examples.
27. References to authentication and access controls are noted in the previous chapter. Privacy-enhanced mail is discussed by Kent [66]. Some of the implications of computer-communication privacy are discussed by Rotenberg [140] and Tuerkheimer [141].
Chapter 7
1. Translation by Stephen Mitchell, Harper and Row, 1988.
2. See SEN 15, 2; 15, 3; 15, 5; see also M.M. Waldrop, Science 249, pp. 735-36, August 17, 1990.
3. Adapted from Inside Risks, Communications of the ACM, 33, 9, 202, September 1990.
4. San Francisco Chronicle, September 4, 1993, p. B1.
5. It appears that this statement is also true of many people, but I persevere nonetheless—perhaps encouraged by Freud’s remark that the true test of maturity is a person’s ability to generate and appreciate humor.
6. This section is drawn on material originally presented in [109].
7. For an excellent article on fault tolerance in distributed systems in general, see Cristian [30].
8. Translation by Stephen Mitchell, Harper and Row, 1988.
9. Contributed by Ross Anderson in RISKS 14, 41, March 12, 1993, from British press reports.
10. See the Richmond Times-Dispatch, June 8, 1987, p. B1, from Nick Condyles, SEN 12, 3.
11. See SEN 12, 4; 13, 1; 13, 2, contributed by Rodney Hoffman, who also contributed the subsequent five items, abstracted from Business Week, November 7, 1988; see SEN 14, 1.
12. See The Washington Times, February 15, 1989, from Joseph M. Beckman in SEN 14, 2.
13. See the Los Angeles Daily News, February 25, 1991, summarized in SEN 16, 2.
14. See Jeff Gerth, The New York Times, December 1, 1989, noted in SEN 15, 1 by Gary Chapman, who also contributed the subsequent 5 items to SEN 15, 1, 21-22.
15. This item was noted by James Paul in SEN 17, 3.
16. James H. Paul and Gregory C. Simon, “Bugs in the System: Problems in Federal Government Computer Software Development and Regulation” (from the Subcommittee on Investigations and Oversight of the House Committee on Science, Space, and Technology, U.S. Government Printing Office, Washington, D.C., September 1989).
17. This quote is included in a summary of the Paul-Simon report by M. Mitchell Waldrop in Science, 246, p. 753, November 10, 1989. See also SEN 15, 1.
18. See Aviation Week, May 27, 1991, and Henry Spencer in SEN 16, 4.
19. From the Minneapolis Star Tribune, p. 1 and 4D, November 28, 1987, excerpted by Scot E. Wilcoxon in SEN 15, 5.
20. The New York Times, July 4, 1983.
21. See Epstein and Smith, “Hartford Roof Failure: Can We Blame the Computer?”, Proceedings of the Seventh Conference on Electrical Computation, 1979, noted by Richard S. D’Ippolito in SEN 11, 5. Peter Desnoyers later reported in SEN 14, 5 that there was only a single part-time weld inspector.
22. Sail Wars was broadcast on NOVA on December 9, 1986; this case was noted in SEN 12, 1 by Bruce Wampler.
23. This section is derived from [111].
24. See the assured-pipeline approach based on the LOgical Coprocessor Kernel, LOCK [13].
25. In the language of state machines, a TCB is basically a nontamperable-type manager for an abstract data type.
26. System-level concepts include dual redundancy as in Tandem’s NonStop systems [53], triple-modular redundancy (TMR) with majority voting as in the Software Implemented Fault Tolerant (SIFT) aircraft flight-control system [18, 19, 91, 98, 175, 176], and combinations such as the dual-dual-plus-separately-programmed-backup quintuple redundancy of the space-shuttle systems. Subsystem-level concepts include techniques for server fault tolerance, such as highly available file servers (ranging from complete but slower coverage at one extreme to possibly incomplete but rapid recovery, as in the Sprite strategy for speedy recovery [7] using local contextual knowledge in a distributed state), and two-phase commits in transaction-based systems to defend against crashes and other unexpected events during updates of distributed data and during the execution of nonatomic transactions. Lower-layer examples include error-correcting and error-detecting codes in memory and communications, instruction retry to overcome transient errors in processing, reliable and efficient broadcast communication protocols (for example, [61, 160]), multiple paths in communications, and retries following failed communications or timeouts. Related techniques are also used for information integrity, such as check sums and cryptoseals. Byzantine algorithms [70] provide an interesting subject of research, but may be impractical except in truly critical situations. (Simplifying assumptions that reduce the complexity and the time to completion have been considered by Flaviu Cristian [29], at the expense of diminished fault-tolerance coverage.) The reader seriously interested in reliability and fault tolerance will find an enormously rich literature, including the proceedings of the Fault Tolerant Computing Conferences and historically interesting volumes such as [3]. There are also many early works of significant historical interest, such as John von Neumann’s paper on building reliable circuits out of unreliable components [173] and a corresponding article by Ed Moore and Claude Shannon [94] for switching circuits built out of what were affectionately referred to as “crummy relays.” An early guide to error-correcting codes is given by Wes Peterson and Ed Weldon [128]; the literature since then has specialized into all sorts of varieties, considering binary and nonbinary codes, random and burst errors, arithmetic codes, and so on. A useful book on error-correcting coding is that by Thammavarapu (T.R.N.) Rao [135].
Additional references relating to techniques for increasing system reliability in response to hardware faults and communications failures are explored (for example) in [11, 38, 56, 69, 76, 107, 125, 134, 157]. The contributions of good software-engineering practice are considered in Section 7.8.
A fundamental survey article on software safety is given by Nancy Leveson [79], whose work on software safety, fault-tree analysis, hazard analysis [80, 82] and fault-tolerant software [68, 154] is particularly relevant for analyzing system safety.
A set of criteria for evaluating system safety is in use in the United Kingdom [169, 170].
27. This section is derived from [112].
28. Electronic Voting System, Request for Proposal, Appendix G, Security and Control Considerations, New York City Board of Elections, New York City Elections Project, September 1987.
29. This statement can also be applied to persons employed by state lotteries.
30. A client-server authentication scheme has been proposed that encrypts seemingly randomized data with a fixed password [52]. This scheme overcomes a potential weakness in Kerberos—namely, that a key derived from a fixed password may be compromisible by off-line guessing. The complexity of this scheme is comparable to the use of cryptographic tokens, and may have advantages over certain token implementations.
31. Robert Formaini [45] considers the risks of risk assessment in establishing public policy. See also books by Barry Boehm [14], Robert Charette [22, 23], and Capers Jones [60], as well as [151].
32. Elucidating her view of minimalism in psychohistory to Hari Seldon, in Forward the Foundation, by Isaac Asimov [5].
33. Various references to specific aspects of software engineering are given in Section 7.8. The reader may wish to delve into one of many comprehensive books on the subject, such as that by Stephen R. Schach [147]. An excellent book on modern operating systems by Andrew Tanenbaum [166] is also suggested. However, the references given here are only a few of those necessary to gain a grasp of the problems of system development.
Chapter 8
1. See articles by Dorothy Denning [31], Peter Denning [33], and Don Norman [117], and books by Norman [116] and Charles Perrow [126]. For some psychosocial implications relating to the development and use of computer systems, see [106], written by the author in response to Robert Pirsig’s Zen and the Art of Motorcycle Maintenance [132].
2. From “Corrupted Polling” (Inside Risks), CACM, 35, 11, November 1992, p. 138.
3. The Washington Post, May 23, 1986.
4. The New York Times, June 2, 1986.
5. Some computer-minded critics have faulted the movie for bearing little resemblance to reality. To be sure, extremely critical and data-sensitive computer systems should generally not permit remote dialup access with easily compromised user authentication. However, the movie’s computer-related effects are in fact based on events that have happened (many of which are noted in this book) — mistaken detections of what appeared to be incoming missiles, such as BMEWS failing to identify the moon properly; capture and use of a code by record-and-playback (in this case, touch-tone telephone signals); computer-based searching for modem-answering telephone numbers; the presence of a hidden and supposedly private trapdoor access path for the convenience of the system developers; the use of an easily guessed password; the presence of game programs (notably tic-tac-toe) in unexpected places and their unexpected interactions with other programs, and so on.
6. The remainder of this section is drawn on material originally presented in [108].
7. See Science, 259, p. 305, January 15, 1993.
8. See also Article 35 of the same book [35], which includes four separately authored pieces on ethics related to computers.
9. The PEM standard [66] has various implementations for message confidentiality and authenticity, including RSA Data Security’s RIPEM.
10. A fascinating collection of ethical conflicts worthy of detailed study is provided by Donn Parker [121]. An important book on computer ethics is authored by Deborah Johnson [59]. Ethical behavior in engineering is considered by Stephen Unger [171]. See also the Proceedings of the National Conference on Computing and Values, August 12 through 16, 1991, New Haven, Connecticut; for information, contact Terry Bynum, Research Center on Computing and Society, Southern Connecticut State Univ., New Haven CT 06515, telephone 1-(203) 397-4423.
An Invitational Conference on Legal, Ethical, and Technological Aspects of Computer and Network Use and Abuse was held on December 17-19, 1993, sponsored by the joint National Conference of Lawyers and Scientists of the American Association for the Advancement of Science (AAAS) and the American Bar Association (ABA). The collection of position papers addresses the subject from a wide range of viewpoints. A second conference in this series was held on October 7-9, 1994. For information, contact Elizabeth Gehman, AAAS, 1333 H St NW, Washington DC 20005, [email protected].
An important book on value conflicts and social choices is edited by Charles Dunlop and Rob Kling; its second edition [67] represents a substantial revision of the first edition, and includes many recent articles.
11. In the years since the precursor of this section originally appeared as the Inside Risks column identified in the previous note, the number of universities offering courses or seminars in social implications of computers and other technologies has increased, and the number of readers of RISKS has expanded enormously, suggesting a marked increase in awareness. SRI has been most gracious in tolerating the RISKS activities. Perhaps this book will help to overcome some of the resistance that Ms. Rosenberg cites.
12. Various approaches to structuring and constraining the development team have been contemplated, such as chief-programmer teams and Harlan Mills’ clean-room approach [93], plus countless development methodologies aimed at helping the managers. See also Fred Brooks’ “Mythical Man Month” [17].
13. See a letter to Congress from Jim Bidzos of RSA (RISKS 12, 37, September 20, 1991), as well as subsequent letters to the National Institute of Standards and Technology (NIST) from Ron Rivest (RISKS 12, 57, October 28, 1991) and from Martin Hellman (RISKS, 12, 63, November 13, 1991).
14. This section is based on the Inside Risks column of the Communications of the ACM, 34, 2, 130, February 1991.
15. For a critical appraisal of today’s bureaucratized computer education in our schools, and for some acerbic recommendations on what might be done in the future, see Seymour Papert, The Children’s Machine: Rethinking School in the Age of the Computer [118]. For another view, see Steve Talbott, The Future Does Not Compute [165].
Chapter 9
1. Display in front of PJ Auto Sales in Revere, Massachusetts, January 1993, observed and contributed by Helen K. Neumann.
2. Henrietta Temple, book II, 1837.
3. See Tom Wolfe, The Right Stuff, 1979, pp. 300-306.
4. UPI, in San Francisco Chronicle, July 31, 1987; SEN 12, 4.
5. See the San Francisco Chronicle, April 10, 1983, p. A5 and SEN 18, 3, A-14.
6. San Francisco Chronicle, July 28, 1993, p. C1.
7. Mostly Harmless, book 5 of the Hitch Hiker’s Guide to the Galaxy trilogy, Heinemann, London, 1992, contributed by Pete Mellor, in SEN 18, 2, 5.
8. Thanks to R. Jagannathan.
9. The remainder of this section is based on the “Inside Risks” column in the Communications of the ACM, 37 6, 114, June 1994.
10. Quoted by Jerome Agel and Walter D. Glanze in Cleopatra’s Nose, The Twinkie Defense & 1500 Other Verbal Shortcuts in Popular Parlance, Prentice Hall Press, Simon and Schuster, 1990.
11. The material in this section is taken from Inside Risks, Communications of the ACM, 36, 3, 130, March, 1993.
12. See the Winter 1991 issue of the Whole Earth Review, which includes a diverse collection of articles on technology by Jerry Mander, Howard Levine, Langdon Winner, Patricia Glass Schuman, Linda Garcia, Gary T. Marx, Ivan Illich, and Amory and Hunter Lovins. See also “a remarkable essay by the eccentric and curmudgeonly fluid dynamicist Clifford Truesdell, “The Computer: Ruin of Science and Threat to Mankind,” in An Idiot’s Fugitive Guide to Science, Springer-Verlag, 1984, pointed out to me by Michael Tobis.
13. Centre for Software Reliability, City University, Northampton Square, London EC IV OHB, England.
14. The number of events recorded here and the volume of space occupied in the RISKS archives during any time interval are not necessarily indicative of the frequency or seriousness of observed problems. Over the years, I have become increasingly selective about what gets included in the on-line RISKS newsgroup and its ensuing published highlights, and have worked harder to make the printed descriptions more incisive. Thus, certain types of recent cases may be somewhat underrepresented in this book. For example, occurrences of cases nearly identical to those already reported would otherwise tend to overwhelm the reader—as in the case of naming problems considered in Section 6.5, personal-computer viruses, and security misuses.
Epilogue
1. Isaac Asimov, “It’s Such a Beautiful Day,” Copyright 1954, Ballantine Books, Inc., reprinted in Isaac Asimov, The Complete Stories, volume 1, A Foundation Book, Doubleday, New York, 1990.