Companies that provide web services for the outside world in their own infrastructure are exposed to a variety of threats. The developers of the open source mitmproxy tool describe it as the Swiss army knife for debugging, testing, data protection analysis, and penetration testing HTTP(S) connections. I show you how mitmproxy can be a useful addition to your security toolbox.

Man in the Middle

When most people hear the term "proxy," they probably think of legacy proxy servers that act as gateways connecting local networks to the global network or as go-betweens protecting local clients against external access (e.g., NGINX, Squid, and WinGate come to mind). Given the name, mitmproxy [1] could be assumed to be in the same category. However, the tool takes a different approach by specializing in HTTP(S) traffic analysis. Like Wireshark, the software is more of a sniffer that records the data traffic between the HTTP client and server and enables analysis by doing so.

The mitm part of the name hints at its functionality: mitmproxy acts as a man-in-the-middle (MITM) proxy that intercepts and modifies HTTP and HTTPS data traffic. You can record the HTTP conversation for later analysis, although the tool is limited to the protocol-specific data exchange. Unlike Wireshark and other sniffers, no other data is logged.

Mitmproxy can also act as a reverse proxy and forward data traffic to a specific server. Script-based manipulation of HTTP traffic is also an option, for which you can use simple Python scripts. Interaction with third-party applications for automatic manipulation or visualization is also possible with the Python API. Mitmproxy can generate SSL/TLS certificates for interception, as well.

Basic Principles

A basic understanding of how mitmproxy works is useful if you want to work effectively in the environment. For example, take a look at HTTPS-protected access by a client to a web server. The client uses the HTTP command:

CONNECT server.en:443 HTTP/1.1

A legacy proxy server cannot manipulate SSL-/TLS-encrypted data traffic but simply forwards the request to the target system; it thus lives up to its name as an authorized agent. When you use mitmproxy, the HTTPS proxy sits between the client and server with the classic man-in-the-middle approach. For the client, mitmproxy looks like a server, while pretending to be the client for the server, which allows mitmproxy to decrypt the data traffic from both sides.

The challenge for mitmproxy is that the certification authority's system is designed to prevent precisely this type of attack by allowing a trusted third party to sign a server's certificates cryptographically to verify its legitimacy. If any discrepancies are noted, the connection is interrupted, which is why it is often difficult to analyze secure connections.

The mitmproxy developers used a trick to solve this problem. The software itself acts as a trustworthy certificate authority (CA). To do so, mitmproxy comes with a complete CA implementation that generates all the required certificates on the fly. For the client to trust these certificates, you need to register mitmproxy manually as a trusted CA.

Mitmproxy then needs to overcome further challenges to inject the environment between the client and server without being noticed. For example, the domain name of the remote party must be determined so that it can be used in the intercept certificate. To do so, mitmproxy uses upstream certificate sniffing. The tool also extracts the common name (CN) from the upstream certificate and the subject alternative names (SANs). It also cleverly works around the server name indication (SNI) handover.

The following processes take place for HTTPS connections with an intermediate mitmproxy (Figure 1):

Figure 1: Mitmproxy's man-in-the-middle position allows web traffic to be intercepted and manipulated.

1. The client establishes a connection to mitmproxy and generates an HTTP CONNECT request.
2. Mitmproxy responds with a 200 Connection Established message, simulating a CONNECT pipe being opened.
3. The client opens an SSL/TLS connection on the assumption that it is talking to the desired server. On doing so, it states the SNI for the hostname.
4. Mitmproxy opens the connection to the server and sets up a secure connection with the SNI hostname specified by the client.
5. The target system responds with a certificate containing the CN and SAN values required to create the interception certificate.
6. Mitmproxy generates the interception certificate and continues the client SSL/TLS handshake interrupted in step 3.
7. The client sends the request through the open connection.
8. Mitmproxy forwards the request to the server through the connection initiated in step 4.

This simplified process sequence shows the sophistication of the actions performed by mitmproxy just to slip into position between the client and the server.

Getting Started

Mitmproxy is available for Linux, macOS, and Windows. Standalone binaries are available for Windows and Linux, with distribution-specific packages for various Linux distributions (e.g., Arch, Debian, Ubuntu, and Kali). After installing on Windows, mitmproxy, mitmdump, and mitmweb are added to PATH and can be called from the command line. In principle, the Linux packages will also run on Windows Subsystem for Linux (WSL).

If you prefer to use a Docker container [2], use the following command to launch the mitmproxy terminal interface:

docker run --rm -it -v ~/.mitmproxy:/home/mitmproxy/.mitmproxy -p 8080:8080 mitmproxy/mitmproxy

To make sure the client's web traffic is routed through mitmproxy, you need to adjust the global settings when installing the software locally and enter a proxy address of 127.0.0.1:8080 . You could also use a proxy auto-configuration (PAC) file to simplify the global network configuration; the required files are available online [3].

If you try to access the Internet with your web browser after configuring the proxy, the attempt will fail. Your browser displays a Your connection is not private error message. The reason is simple: The browser does not consider the intermediary proxy to be trustworthy because it considers the mitmproxy certificate invalid. Once the proxy server is activated, you can pick up the required certificate from the mitm.it website. Valid certificates for all supported platforms are available there. On Windows, use the import wizard to deploy the certificate. On Linux, the easiest way to import the certificate is with the command:

sudo security add-trusted-cert -d -p ssl -p basic -k /Library/Keychains/System.keychain mitmproxy-ca-cert.pem

Installation instructions are available online [4] for all other platforms. You can then view and, if needed, edit the details in the certificate manager belonging to your choice of operating system.