Mitmproxy is an interactive console program that allows traffic flows to be intercepted, inspected, modified, and replayed. After researching for this section, I decided that the easiest and most complete way to learn about HTTP proxies in Python is by using mitmproxy. Any other attempt is more complex and limited than mitmproxy.
Mitmproxy is developed in Python and allows users to extend it via their inline scripts. It supports SSL out of the box, unlike other alternatives out there that support only HTTP.
Let's see how mitmproxy works using a simple example. If we go to the Terminal and type mitmproxy, we get an mitmproxy console listening at port 8080:
If we change our browser settings to use proxy for HTTP connections, and we fire up a request such as http://www.edge-security.com/, we will see all the requests in the console.
Let's click on the Open menu icon on the right-hand side of the browser and go to Preferences | Advanced | Network | Connection | Settings... | Manual proxy configuration. Set the HTTP Proxy as 127.0.0.1 and the Port as 8080 and hit OK:
Let's load http://www.edge-security.com/ in the browser now; you can see the request history in the console:
Now, if you select a request and press Enter, you will see the details for that request, the response, the headers, and the connection details. If you want to edit the request, press E. When done, you can send it by pressing R. This is the basic usage of mitmproxy.
Just to remind you how proxies work, in this particular case I have set up my browser to connect to mitmproxy on the localhost in port 8080. The browser and the proxy are on the same machine:
Mitmproxy has a powerful scripting API that will let us access requests on the fly to manipulate them:
The mitm scripting API is event-driven and its script is simply a Python module that exposes a set of event methods.
We can see in the screenshot an example of a simple inline script that will add a new header to every HTTP response before it is returned to the client. That is all the code needed for the script. In the next section, we're going to learn how to write a script to manipulate requests in mitmproxy.