Web crawling is the process of getting specific information from websites using a bot or automated script. Kali provides the inbuilt applications to perform this activity. The benefit of web crawling is that it lets you scrape data without having to perform attacks manually, one by one.

Attackers can use WebSploit to perform the web scan and crawling, and also to analyze the web. For example, to identify the phpmyadmin on multiple sites, attackers can configure the WebSploit module by running WebSploit in the Terminal, typing use web/pma, setting the target host using set target victim, and running it, as shown in the following screenshot:

Attackers can also make use of OWASP, DirBuster, and other tools to perform the same actions.