In the previous section, we defined a simple inline script in order to access the response from a request. Other parts of the communication, the mitmproxy, let us access the response via handlers:

• start: This is called once the script starts up, before any other events
• clientconnect : This is called when a client initiates a connection to the proxy

• request: This is called when a client request has been received
• serverconnect : This is called when the proxy initiates a connection to the target server
• responseheaders: This is called when the responseheaders for a server response have been received, but the response body has not been processed
• response: This is called when a server response has been received
• error: This is called when a flow error has occurred
• clientdisconnect: This is called when a client disconnects from the proxy
• done: This is called when the script shuts down after all other events

So now that we know what handlers are available to us, let's look at an example where we access a request.

Let's open the mitm-0.py script, located in the source code for Section-7, in the editor. This script will basically log every request that the proxy receives from the client.

We can see that this script is very simple:

import sys

def request(context, flow):
  f = open('httplogs.txt', 'a+')
  f.write(flow.request.url + '
') 
  f.close()

We have the handler for the request, with the mandatory first parameter called context and the second being flow. flow, as the name suggests, holds all the information about the communication. In the function, we open the httplogs.txt file, then we write flow.request.url, which is the URL requested by the client, and we finally close the f file.

Let's go back to the Terminal in the Section-7 directory. Type mitmproxy -s mitm-0.py and the mitmproxy console will come up. Then, we will open our browser and change the proxy configuration so it points to localhost 8080. Click on the Open menu icon at the right-hand side of the browser and go to Preferences | Advanced | Network | Connection | Settings... | Manual proxy configuration. Set the Port as 8080. Remove localhost and 127.0.0.1 from No Proxy for:

Let's load www.scruffybank.com in the browser. You can see all the requests in the console:

Let's close the console and view the httplogs.txt file. We can open it with the editor. We can see all URLs requested in the session:

Excellent work!

Now, let's add a filter to log unique URLs to avoid storing duplicate URLs. Open the mitm-1.py file in the editor. In order to prevent duplicates, let's create a global variable in the script called history; then, in the function, we just check that the URL is not in the history:

import sys

global history 
history = [] 

def request(context, flow):
  global history
  url = flow.request.url
  if url not in history:
    f = open('httplogs.txt', 'a+')
    f.write(flow.request.url + '
') 
    f.close()
    history.append(url)
  else:
    pass

If not present, we log it and then we add it to the history. Let's try it again and see if it works. First, we can remove the httplogs.txt file by right-clicking on it and selecting the Delete option. Run mitmproxy -s mitm-1.py.

Let's go back to the browser and open www.scruffybank.com/login.php and refresh it several times. Close the proxy console, and open the results again:

Great! No duplicates.

Now that we know how to access requests, let's see how we can add a query string parameter to every request. You may ask why. Well, we need to add certain parameters in a request in order to access certain information.

Let's open mitm-2.py in the editor. Now, what we're doing is getting the query string with flow.request.get_query(), and then we're checking whether the query string has some content:

import sys

def request(context, flow):
  q = flow.request.get_query()
  if q:
    q["isadmin"] = ["True"]
    flow.request.set_query(q)

If there is content, we add a new parameter called isadmin with the value True. And finally, we update the request query string with flow.request.set_query(q).

Let's try it in the command line. Let's launch mitm-2.py by typing mitmproxy -s mitm-2.py. In the browser, click on the Learn More link, which has parameters.

In the mitmproxy console, you can see that mitmproxy is adding the isadmin query string parameter with the True value:

In this case, it won't do anything, but it is a warm-up for the next section where we're going to learn how to do something more complex such as testing SQLi for every parameter we see in the proxy.