- Webbots, Spiders, and Screen Scrapers
- ACKNOWLEDGMENTS
- Introduction
- I. FUNDAMENTAL CONCEPTS AND TECHNIQUES
- 1. WHAT'S IN IT FOR YOU?
- 2. IDEAS FOR WEBBOT PROJECTS
- 3. DOWNLOADING WEB PAGES
- 4. PARSING TECHNIQUES
- 5. AUTOMATING FORM SUBMISSION
- 6. MANAGING LARGE AMOUNTS OF DATA
- II. PROJECTS
- 7. PRICE-MONITORING WEBBOTS
- 8. IMAGE-CAPTURING WEBBOTS
- 9. LINK-VERIFICATION WEBBOTS
- 10. ANONYMOUS BROWSING WEBBOTS
- 11. SEARCH-RANKING WEBBOTS
- 12. AGGREGATION WEBBOTS
- 13. FTP WEBBOTS
- 14. NNTP NEWS WEBBOTS
- 15. WEBBOTS THAT READ EMAIL
- 16. WEBBOTS THAT SEND EMAIL
- 17. CONVERTING A WEBSITE INTO A FUNCTION
- III. ADVANCED TECHNICAL CONSIDERATIONS
- 18. SPIDERS
- 19. PROCUREMENT WEBBOTS AND SNIPERS
- 20. WEBBOTS AND CRYPTOGRAPHY
- 21. AUTHENTICATION
- 22. ADVANCED COOKIE MANAGEMENT
- 23. SCHEDULING WEBBOTS AND SPIDERS
- IV. LARGER CONSIDERATIONS
- 24. DESIGNING STEALTHY WEBBOTS AND SPIDERS
- 25. WRITING FAULT-TOLERANT WEBBOTS
- 26. DESIGNING WEBBOT-FRIENDLY WEBSITES
- 27. KILLING SPIDERS
- 28. KEEPING WEBBOTS OUT OF TROUBLE
- A. PHP/CURL REFERENCE
- Creating a Minimal PHP/CURL Session
- Initiating PHP/CURL Sessions
- Setting PHP/CURL Options
- CURLOPT_URL
- CURLOPT_RETURNTRANSFER
- CURLOPT_REFERER
- CURLOPT_FOLLOWLOCATION and CURLOPT_MAXREDIRS
- CURLOPT_USERAGENT
- CURLOPT_NOBODY and CURLOPT_HEADER
- CURLOPT_TIMEOUT
- CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR
- CURLOPT_HTTPHEADER
- CURLOPT_SSL_VERIFYPEER
- CURLOPT_USERPWD and CURLOPT_UNRESTRICTED_AUTH
- CURLOPT_POST and CURLOPT_POSTFIELDS
- CURLOPT_VERBOSE
- CURLOPT_PORT
- Executing the PHP/CURL Command
- Closing PHP/CURL Sessions
- B. STATUS CODES
- C. SMS EMAIL ADDRESSES
- About the Author
- Colophon
This appendix contains status codes returned by web (HTTP) and news (NNTP) servers. Your webbots and spiders should use these status codes to determine the success or failure communicating with servers. When debug-ging your scripts, status codes also provide hints as to what's wrong.
The following is a representative sample of HTTP codes. These codes reflect the status of an HTTP (web page) request. You'll see these codes returned in $returned_web_page['STATUS'][ 'http_code'] if you're using the LIB_http library.
100 Continue 101 Switching Protocols 200 OK 201 Created 202 Accepted 203 Non-Authoritative Information 204 No Content 205 Reset Content 206 Partial Content 300 Multiple Choices 301 Moved Permanently 302 Found 303 See Other 304 Not Modified 305 Use Proxy 306 (Unused) 307 Temporary Redirect 400 Bad Request 401 Unauthorized 402 Payment Required 403 Forbidden 404 Not Found 405 Method Not Allowed 406 Not Acceptable 407 Proxy Authentication Required 408 Request Timeout 409 Conflict 410 Gone 411 Length Required 412 Precondition Failed 413 Request Entity Too Large 414 Request-URI Too Long 415 Unsupported Media Type 416 Requested Range Not Satisfiable 417 Expectation Failed 500 Internal Server Error 501 Not Implemented 502 Bad Gateway 503 Service Unavailable 504 Gateway Timeout 505 HTTP Version Not Supported
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.