Web scraping is now widely used to get data from websites. Whether it be e-mails, contact information, or selling prices of items, we rely on web scraping techniques as they allow us to collect large data with minimal effort, and also, we don't require database or other backend access to get this data as they are represented as web pages.
Beautiful Soup allows us to get data from HTML and XML pages. This book helps us by explaining the installation and creation of a sample website scraper using Beautiful Soup. Searching and navigation methods are explained with the help of simple examples, screenshots, and code samples in this book. The different parser support offered by Beautiful Soup, supports for scraping pages with encodings, formatting the output, and other tasks related to scraping a page are all explained in detail. Apart from these, practical approaches to understanding patterns on a page, using the developer tools in browsers will enable you to write similar scrapers for any other website.
Also, the practical approach followed in this book will help you to design a simple web scraper to scrape and compare the selling prices of various books from three websites, namely, Amazon, Barnes and Noble, and PacktPub.
Chapter 1, Installing Beautiful Soup, covers installing Beautiful Soup 4 on Windows, Linux, and Mac OS, and verifying the installation.
Chapter 2, Creating a BeautifulSoup Object, describes creating a BeautifulSoup object from a string, file, and web page; discusses different objects such as Tag, NavigableString, and parser support; and specifies parsers that scrape XML too.
Chapter 3, Search Using Beautiful Soup, discusses in detail the different search methods in Beautiful Soup, namely, find(), find_all(), find_next(), and find_parents(); code examples for a scraper using search methods to get information from a website; and understanding the application of search methods in combination.
Chapter 4, Navigation Using Beautiful Soup, discusses in detail the different navigation methods provided by Beautiful Soup, methods specific to navigating downwards and upwards, and sideways, to the previous and next elements of the HTML tree.
Chapter 5, Modifying Content Using Beautiful Soup, discusses modifying the HTML tree using Beautiful Soup, and the creation and deletion of HTML tags. Altering the HTML tag attributes is also covered with the help of simple examples.
Chapter 6, Encoding Support in Beautiful Soup, discusses the encoding support in Beautiful Soup, creating a BeautifulSoup object for a page with specific encoding, and the encoding supports for output.
Chapter 7, Output in Beautiful Soup, discusses formatted and unformatted printing support in Beautiful Soup, specifications of different formatters to format the output, and getting just text from an HTML page.
Chapter 8, Creating a Web Scraper, discusses creating a web scraper for three websites, namely, Amazon, Barnes and Noble, and PacktPub, to get the book selling price based on ISBN. Searching and navigation methods used to create the parser, use of developer tools so as to identify the patterns required to create the parser, and the full code sample for scraping the mentioned websites are also explained in this chapter.