- Webbots, Spiders, and Screen Scrapers
- ACKNOWLEDGMENTS
- Introduction
- I. FUNDAMENTAL CONCEPTS AND TECHNIQUES
- 1. WHAT'S IN IT FOR YOU?
- 2. IDEAS FOR WEBBOT PROJECTS
- 3. DOWNLOADING WEB PAGES
- 4. PARSING TECHNIQUES
- 5. AUTOMATING FORM SUBMISSION
- 6. MANAGING LARGE AMOUNTS OF DATA
- II. PROJECTS
- 7. PRICE-MONITORING WEBBOTS
- 8. IMAGE-CAPTURING WEBBOTS
- 9. LINK-VERIFICATION WEBBOTS
- 10. ANONYMOUS BROWSING WEBBOTS
- 11. SEARCH-RANKING WEBBOTS
- 12. AGGREGATION WEBBOTS
- 13. FTP WEBBOTS
- 14. NNTP NEWS WEBBOTS
- 15. WEBBOTS THAT READ EMAIL
- 16. WEBBOTS THAT SEND EMAIL
- 17. CONVERTING A WEBSITE INTO A FUNCTION
- III. ADVANCED TECHNICAL CONSIDERATIONS
- 18. SPIDERS
- 19. PROCUREMENT WEBBOTS AND SNIPERS
- 20. WEBBOTS AND CRYPTOGRAPHY
- 21. AUTHENTICATION
- 22. ADVANCED COOKIE MANAGEMENT
- 23. SCHEDULING WEBBOTS AND SPIDERS
- IV. LARGER CONSIDERATIONS
- 24. DESIGNING STEALTHY WEBBOTS AND SPIDERS
- 25. WRITING FAULT-TOLERANT WEBBOTS
- 26. DESIGNING WEBBOT-FRIENDLY WEBSITES
- 27. KILLING SPIDERS
- 28. KEEPING WEBBOTS OUT OF TROUBLE
- A. PHP/CURL REFERENCE
- Creating a Minimal PHP/CURL Session
- Initiating PHP/CURL Sessions
- Setting PHP/CURL Options
- CURLOPT_URL
- CURLOPT_RETURNTRANSFER
- CURLOPT_REFERER
- CURLOPT_FOLLOWLOCATION and CURLOPT_MAXREDIRS
- CURLOPT_USERAGENT
- CURLOPT_NOBODY and CURLOPT_HEADER
- CURLOPT_TIMEOUT
- CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR
- CURLOPT_HTTPHEADER
- CURLOPT_SSL_VERIFYPEER
- CURLOPT_USERPWD and CURLOPT_UNRESTRICTED_AUTH
- CURLOPT_POST and CURLOPT_POSTFIELDS
- CURLOPT_VERBOSE
- CURLOPT_PORT
- Executing the PHP/CURL Command
- Closing PHP/CURL Sessions
- B. STATUS CODES
- C. SMS EMAIL ADDRESSES
- About the Author
- Colophon
Now that you know how to use webbots to interface with newsgroups, here is a list of ideas you can use to develop news bots for your own purposes.
Develop a newsgroup clipping service. This service could monitor numerous newsgroups for mention of specific keywords and either aggregate that information in a database or send email alerts when a keyword appears in a newsgroup.
Build a web-based newsgroup portal, similar to http://groups.google.com.
Create a webbot that gathers weather forecasts for Las Vegas from the National Weather Service website, and post this weather information for vacationers on
alt.vacation.las-vegas.[48]Monitor newsgroups for unauthorized use of intellectual property.
Create a database that archives a newsgroup.
Write a web-based newsgroup client that allows users to read newsgroups anonymously.
[48] Due to the ridiculous amounts of spam on newsgroups, scripts for posting articles on newsgroups were deliberately omitted from this chapter. However, between the scripts used as examples in this chapter and the original NNTP RFC, you should be able to figure out how to post articles to newsgroups on your own.
-
No Comment