- Webbots, Spiders, and Screen Scrapers
- ACKNOWLEDGMENTS
- Introduction
- I. FUNDAMENTAL CONCEPTS AND TECHNIQUES
- 1. WHAT'S IN IT FOR YOU?
- 2. IDEAS FOR WEBBOT PROJECTS
- 3. DOWNLOADING WEB PAGES
- 4. PARSING TECHNIQUES
- 5. AUTOMATING FORM SUBMISSION
- 6. MANAGING LARGE AMOUNTS OF DATA
- II. PROJECTS
- 7. PRICE-MONITORING WEBBOTS
- 8. IMAGE-CAPTURING WEBBOTS
- 9. LINK-VERIFICATION WEBBOTS
- 10. ANONYMOUS BROWSING WEBBOTS
- 11. SEARCH-RANKING WEBBOTS
- 12. AGGREGATION WEBBOTS
- 13. FTP WEBBOTS
- 14. NNTP NEWS WEBBOTS
- 15. WEBBOTS THAT READ EMAIL
- 16. WEBBOTS THAT SEND EMAIL
- 17. CONVERTING A WEBSITE INTO A FUNCTION
- III. ADVANCED TECHNICAL CONSIDERATIONS
- 18. SPIDERS
- 19. PROCUREMENT WEBBOTS AND SNIPERS
- 20. WEBBOTS AND CRYPTOGRAPHY
- 21. AUTHENTICATION
- 22. ADVANCED COOKIE MANAGEMENT
- 23. SCHEDULING WEBBOTS AND SPIDERS
- IV. LARGER CONSIDERATIONS
- 24. DESIGNING STEALTHY WEBBOTS AND SPIDERS
- 25. WRITING FAULT-TOLERANT WEBBOTS
- 26. DESIGNING WEBBOT-FRIENDLY WEBSITES
- 27. KILLING SPIDERS
- 28. KEEPING WEBBOTS OUT OF TROUBLE
- A. PHP/CURL REFERENCE
- Creating a Minimal PHP/CURL Session
- Initiating PHP/CURL Sessions
- Setting PHP/CURL Options
- CURLOPT_URL
- CURLOPT_RETURNTRANSFER
- CURLOPT_REFERER
- CURLOPT_FOLLOWLOCATION and CURLOPT_MAXREDIRS
- CURLOPT_USERAGENT
- CURLOPT_NOBODY and CURLOPT_HEADER
- CURLOPT_TIMEOUT
- CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR
- CURLOPT_HTTPHEADER
- CURLOPT_SSL_VERIFYPEER
- CURLOPT_USERPWD and CURLOPT_UNRESTRICTED_AUTH
- CURLOPT_POST and CURLOPT_POSTFIELDS
- CURLOPT_VERBOSE
- CURLOPT_PORT
- Executing the PHP/CURL Command
- Closing PHP/CURL Sessions
- B. STATUS CODES
- C. SMS EMAIL ADDRESSES
- About the Author
- Colophon
Since FTP is often the only application-level protocol that computer systems share, it is a convenient communication bridge between new and old computer systems. Moreover, in addition to using FTP as a common path between disparate—or obsolete—systems, FTP is still the most common method for uploading files to websites. With the information in this chapter, you should be able to write webbots that update websites with information found at a variety of sources. Here are some ideas to get you started.
Write a webbot that updates your corporate server with information gathered from sales reports.
Develop a security webbot that uses a webcam to take pictures of your warehouse or parking lot, timestamps the images, and uploads the pictures to an archival server.
Design a webbot that creates archives of your company's internal forums on an FTP server.
Create a webbot that photographically logs the progress of a construction site and uploads these pictures to an FTP server. Once construction is complete, compile the individual photos into an animation showing the construction process.
If you don't have access to an FTP server on the Internet, you can still experiment with FTP bots. An FTP server is probably already on your computer if your operating system is Unix, Linux, or Mac OS X. If you have a Windows computer, you can find free FTP servers on many shareware sites. Once you locate FTP server software, you can set up your own local server by following the instructions accompanying your FTP installation.
-
No Comment