In this chapter, we saw what web application mapping is. We learned how to create a basic web application crawler. In this chapter, we added recursion capabilities and also learned how to make our crawler recursive.

Finally, we learned how to develop a web application crawler using Python and the Scrapy library. This will be useful for mapping the web application structure and to harvest interesting information such as forms, emails, and comments from the source code of the pages.

Now, we know how to map a web application using a crawler, but most of the applications have hidden resources. These resources are not accessible for all the users or are not linked by all. Luckily, we can use the brute force technique to discover directories, files, or parameters in order to find vulnerabilities or interesting information that we can use in our tests.

In Chapter 4, Resources Discovery, we'll write a tool to perform brute force attacks in different parts of the web application.