To continue our information-gathering phase in website penetration testing, we'll attempt to discover any sensitive files and directories on a target website. To perform this task, we will be using DirBuster. DirBuster is a brute force web application that was designed with the objective of revealing any sensitive directories and files on a target web server.

For this exercise, we'll be using the OWASP Broken Web Applications (BWA) Project virtual machine as our target, and our Kali Linux machine as the attacker.

To discover sensitive files on a web server, follow these steps:

1. Open DirBuster by navigating to Applications | 03 – Web Application Analysis | Web Crawlers & Directory Bruteforcing | DirBuster.
2. When DirBuster opens, enter the IP address of the OWASP BWA virtual machine in the Target URL field. The URL should be in the http://192.168.56.101:80/ format.
3. Optionally, you can increase the number of threads. Increasing the number of threads will apply more computing power to the application and will, therefore, speed up the process.
4. Click on Browse to add a wordlist that DirBuster will use to index and search on the target website. If you click on List Info, a new window will appear, providing a recommended wordlist.

5. Uncheck the box next to Be Recursive.
6. Click on Start to begin the process.

The following screenshot shows the options that were used for this task:

Additionally, you can use a wordlist from another location, such as SecLists.

While DirBuster is performing its brute force attack, the results window will appear. To view all the current directories and files, click on the Results – List View tab, as shown in the following screenshot:

The HTTP 200 Status code indicates that this was successful. In other words, the attacker machine has successfully been able to communicate with a specific directory on the target website/server.

As seen in the previous snippet, the list of directories was found using DirBuster. Go through each directory as they may contain sensitive files and information about the target.

In the next section, we will take a look at the importance of the robots.txt file.