A penetration test can be performed using different approaches, such as Black, Grey, and White box. A Black box test is performed when the testing team doesn't have any previous information about the application to test other than the URL of the server. A White box test is performed when the team has all the information about the target, its infrastructure, software versions, test users, development information, and so on; a Gray box test is intermediate to the Black and White box tests.

For both Black and Gray box tests, a reconnaissance phase is necessary for the testing team to discover the information that is usually provided by the application's owner in a White box approach.

We are going to follow the Black box approach, as it is the one that covers all the steps an external attacker takes to gain enough information in order to compromise certain functions of the application or server.

As a part of every reconnaissance phase in a web penetration test, we will need to browse every link included in a web page and keep a record of every file displayed by it. There are tools that help us automate and accelerate this task called web crawlers or web spiders. These tools browse a web page by following all the links and references to external files, sometimes filling forms and sending them to servers, saving all the requests and responses made, thus giving us the opportunity to analyze them offline.

In this chapter, we will cover the use of some crawlers included in Kali Linux and will also look at the files and directories that will be interesting to look for in a common web page.