A penetration test may be performed using different approaches called black, grey, and white box. Black box is when the testing team doesn't have any previous information about the application to test except the URL of the server; white box is when the team has all information about the target, its infrastructure, software versions, test users, development information, and so on; and gray box is a point in between.

For both black and gray box approaches, a reconnaissance phase, as we saw in the previous chapter, is necessary for the testing team to discover the information that could be provided by the application's owner in a white box approach.

Continuing with the reconnaissance phase in a web penetration test, we will need to browse every link included in a web page and have a record of every file displayed by it. There are tools that help us to automate and accelerate this task; they are called web crawlers or web spiders. These tools browse a web page following all links and references to external files, sometimes filling in forms and sending them to servers, saving all requests and responses made and giving us the opportunity to analyze them offline.

In this chapter, we will cover the use of some proxies, spiders, and crawlers included in Kali Linux and will also see what files and directories would be interesting to look for in a common web page.