The examples built so far work, but each form requires a fair amount of work and testing. This effort can be minimized by using Mechanize, which provides a high-level interface to interact with forms. Mechanize can be installed via pip using this command:

pip install mechanize

Here is how to implement the previous population increase example with Mechanize:

>>> import mechanize
>>> br = mechanize.Browser()
>>> br.open(LOGIN_URL)
>>> br.select_form(nr=0)
>>> br['email'] = LOGIN_EMAIL
>>> br['password'] = LOGIN_PASSWORD
>>> response = br.submit()
>>> br.open(COUNTRY_URL)
>>> br.select_form(nr=0)
>>> br['population'] = str(int(br['population']) + 1)
>>> br.submit()

This code is much simpler than the previous example because we no longer need to manage cookies and the form inputs are easily accessible. This script first creates the Mechanize browser object, and then we navigate to the login URL and select the login form. The selected form inputs are set by passing the name and values directly to the browser object. When debugging, br.form can be called to show the state of a form before it is submitted, as shown here:

>>> print br.form
<POST http://example.webscraping.com/user/login# application/x-www-form-urlencoded
  <TextControl(email=)>
  <PasswordControl(password=)>
  <CheckboxControl(remember=[on])>
  <SubmitControl(<None>=Login) (readonly)>
  <SubmitButtonControl(<None>=) (readonly)>
  <HiddenControl(_next=/) (readonly)>
  <HiddenControl(_formkey=5fa268b4-0dfd-4e3f-a274-e73c6b7ce584) (readonly)>
  <HiddenControl(_formname=login) (readonly)>>

Once the login form parameters are set, br.submit() is called to submit the selected login form, and the script will be logged in. Now, we can navigate to the edit country page, and use the form interface again to increment the population. Note that the population input expects a string, otherwise Mechanize will raise a TypeError exception.

To check whether this has worked, we can use Mechanize to return to the country form and query the current population, as follows:

>>> br.open(COUNTRY_URL)
>>> br.select_form(nr=0)
>>> br['population']
62348449

As expected, the population has increased again and is now 62,348,449.