In this chapter, we saw how we can write a custom crawler from scratch. This task is made easier using Python's modules, such as requests, BeautifulSoup, and so on. Feel free to download the whole code base and test the crawler with various other websites in order to examine its coverage. There may be occasions in which the crawler does not give 100% coverage. Take a look and see for yourself the limitations of the crawler and how it can be improved.
- Title Page
- Copyright and Credits
- Dedication
- About Packt
- Contributors
- Preface
- Introduction to Python
- Technical requirements
- Why Python?
- Getting started
- Python data types
- Numbers
- String types
- String indexes
- String operations through methods and built-in functions
- The replace( ) method
- Substrings or string slicing
- String concatenation and replication
- The strip(), lstrip(), and rstrip() methods
- The split() method
- The find(), index(), upper(), lower(), len(), and count() methods
- The in and not in methods
- The endswith(), isdigit(), isalpha(), islower(), isupper(), and capitalize() methods
- List types
- Tuples in Python
- Dictionaries in Python
- Python operators
- Summary
- Questions
- Building Python Scripts
- Concept Handling
- Advanced Python Modules
- Vulnerability Scanner Python - Part 1
- Vulnerability Scanner Python - Part 2
- Architectural overview
- A closer look at the code
- Driver_scanner.py
- driver_meta.py
- auto_commands.py
- Pexpect – automating terminal: <SSH, Telnet, Wireshark, w3af>
- custom_meta() – automating Metasploit
- singleLineCommands_Timeout() – automating Java , Ruby, Perl, NSE, Python, Bash scripts
- general_interactive() – automating interactive terminal scripts (test_ssl.sh)
- generalCommands_Tout_Sniff() – automating Tshark
- HTTP_based() – automating web specific use-cases
- Storing details in database
- Executing the code
- Database schema for the service-scanning portion of the vulnerability scanner
- GUI version of vulnerability scanner
- A closer look at the code
- Usage [PTO-GUI]
-  Pausing and resuming scans
- Reporting 
- Summary
- Questions
- Further reading
- Architectural overview
- Machine Learning and Cybersecurity
- Automating Web Application Scanning - Part 1
- Automated Web Application Scanning - Part 2
- Building a Custom Crawler
- Reverse Engineering Linux Applications
- Reverse Engineering Windows Applications
- Exploit Development
- Cyber Threat Intelligence
- Other Wonders of Python
- Assessments
- Chapter 1, Introduction to Python
- Chapter 2, Building Python Scripts
- Chapter 3, Concept Handling
- Chapter 4, Advanced Python Modules
- Chapter 5, Vulnerability Scanner Python - Part 1
- Chapter 6, Vulnerability Scanner Python - Part 2
- Chapter 7, Machine Learning and Cybersecurity
- Chapter 8, Automating Web Application Scanning - Part 1
- Chapter 9, Automating Web Application Scanning - Part 2
- Chapter 10, Building a Custom Crawler
- Chapter 11, Reverse Engineering Linux Applications
- Chapter 12, Reverse Engineering Windows Applications
- Chapter 13, Exploit Development
- Chapter 14, Cyber Threat Intelligence
- Chapter 15, Other Wonders of Python
- Other Books You May Enjoy
Summary
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.