- Webbots, Spiders, and Screen Scrapers
- ACKNOWLEDGMENTS
- Introduction
- I. FUNDAMENTAL CONCEPTS AND TECHNIQUES
- 1. WHAT'S IN IT FOR YOU?
- 2. IDEAS FOR WEBBOT PROJECTS
- 3. DOWNLOADING WEB PAGES
- 4. PARSING TECHNIQUES
- 5. AUTOMATING FORM SUBMISSION
- 6. MANAGING LARGE AMOUNTS OF DATA
- II. PROJECTS
- 7. PRICE-MONITORING WEBBOTS
- 8. IMAGE-CAPTURING WEBBOTS
- 9. LINK-VERIFICATION WEBBOTS
- 10. ANONYMOUS BROWSING WEBBOTS
- 11. SEARCH-RANKING WEBBOTS
- 12. AGGREGATION WEBBOTS
- 13. FTP WEBBOTS
- 14. NNTP NEWS WEBBOTS
- 15. WEBBOTS THAT READ EMAIL
- 16. WEBBOTS THAT SEND EMAIL
- 17. CONVERTING A WEBSITE INTO A FUNCTION
- III. ADVANCED TECHNICAL CONSIDERATIONS
- 18. SPIDERS
- 19. PROCUREMENT WEBBOTS AND SNIPERS
- 20. WEBBOTS AND CRYPTOGRAPHY
- 21. AUTHENTICATION
- 22. ADVANCED COOKIE MANAGEMENT
- 23. SCHEDULING WEBBOTS AND SPIDERS
- IV. LARGER CONSIDERATIONS
- 24. DESIGNING STEALTHY WEBBOTS AND SPIDERS
- 25. WRITING FAULT-TOLERANT WEBBOTS
- 26. DESIGNING WEBBOT-FRIENDLY WEBSITES
- 27. KILLING SPIDERS
- 28. KEEPING WEBBOTS OUT OF TROUBLE
- A. PHP/CURL REFERENCE
- Creating a Minimal PHP/CURL Session
- Initiating PHP/CURL Sessions
- Setting PHP/CURL Options
- CURLOPT_URL
- CURLOPT_RETURNTRANSFER
- CURLOPT_REFERER
- CURLOPT_FOLLOWLOCATION and CURLOPT_MAXREDIRS
- CURLOPT_USERAGENT
- CURLOPT_NOBODY and CURLOPT_HEADER
- CURLOPT_TIMEOUT
- CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR
- CURLOPT_HTTPHEADER
- CURLOPT_SSL_VERIFYPEER
- CURLOPT_USERPWD and CURLOPT_UNRESTRICTED_AUTH
- CURLOPT_POST and CURLOPT_POSTFIELDS
- CURLOPT_VERBOSE
- CURLOPT_PORT
- Executing the PHP/CURL Command
- Closing PHP/CURL Sessions
- B. STATUS CODES
- C. SMS EMAIL ADDRESSES
- About the Author
- Colophon
You can point this webbot at any web page, and it will generate a copy of each image that page uses, arranged in a directory structure that resembles the original. You can also develop other useful webbots based on this design. If you want to test your skills, consider the following challenges.
Write a similar webbot that detects hijacked images.
Improve the efficiency of the script by reworking it so that it doesn't download an image it has downloaded previously.
Modify this webbot to create local backup copies of web pages.
Adjust the webbot to cache movies or audio files instead of images.
Modify the bot to monitor when images change on a web page.
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.