During web browsing, we submit web forms many times in a day. Now, you would like do that using the Python code.
This recipe uses a third-party Python module called requests. You can install the compatible version of this module by following the instructions from http://docs.python-requests.org/en/latest/user/install/. For example, you can use pip to install requests from the command line as follows:
$ pip install requests
Let us submit some fake data to register with www.twitter.com. Each form submission has two methods: GET and POST. The less sensitive data, for example, search queries, are usually submitted by GET and the more sensitive data is sent via the POST method. Let us try submitting data with both of them.
Listing 4.4 explains the submit web forms, as follows:
#!/usr/bin/env python # Python Network Programming Cookbook -- Chapter – 4 # This program is optimized for Python 2.7. # It may run on any other version with/without modifications. import requests import urllib import urllib2 ID_USERNAME = 'signup-user-name' ID_EMAIL = 'signup-user-email' ID_PASSWORD = 'signup-user-password' USERNAME = 'username' EMAIL = '[email protected]' PASSWORD = 'yourpassword' SIGNUP_URL = 'https://twitter.com/account/create' def submit_form(): """Submit a form""" payload = {ID_USERNAME : USERNAME, ID_EMAIL : EMAIL, ID_PASSWORD : PASSWORD,} # make a get request resp = requests.get(SIGNUP_URL) print "Response to GET request: %s" %resp.content # send POST request resp = requests.post(SIGNUP_URL, payload) print "Headers from a POST request response: %s" %resp.headers #print "HTML Response: %s" %resp.read() if __name__ == '__main__': submit_form()
If you run this script, you will see the following output:
$ python 4_4_submit_web_form.py Response to GET request: <?xml version="1.0" encoding="UTF-8"?> <hash> <error>This method requires a POST.</error> <request>/account/create</request> </hash> Headers from a POST request response: {'status': '200 OK', 'content- length': '21064', 'set-cookie': '_twitter_sess=BAh7CD-- d2865d40d1365eeb2175559dc5e6b99f64ea39ff; domain=.twitter.com; path=/; HttpOnly', 'expires': 'Tue, 31 Mar 1981 05:00:00 GMT', 'vary': 'Accept-Encoding', 'last-modified': 'Sun, 05 May 2013 15:59:27 GMT', 'pragma': 'no-cache', 'date': 'Sun, 05 May 2013 15:59:27 GMT', 'x-xss-protection': '1; mode=block', 'x-transaction': 'a4b425eda23b5312', 'content-encoding': 'gzip', 'strict-transport- security': 'max-age=631138519', 'server': 'tfe', 'x-mid': 'f7cde9a3f3d111310427116adc90bf3e8c95e868', 'x-runtime': '0.09969', 'etag': '"7af6f92a7f7b4d37a6454caa6094071d"', 'cache-control': 'no- cache, no-store, must-revalidate, pre-check=0, post-check=0', 'x- frame-options': 'SAMEORIGIN', 'content-type': 'text/html; charset=utf-8'}
This recipe uses a third-party module, requests. It has convenient wrapper methods, get() and post(), that do the URL encoding of data and submit forms properly.
In this recipe, we created a data payload with a username, password, and e-mail for creating the Twitter account. When we first submit the form with the GET method, the Twitter website returns an error saying that the page only supports POST. After we submit the data with POST, the page processes it. We can confirm this from the header data.