Web GUI

You can now track the exact HTTP commands in the mitmproxy console (Figure 2), but analyzing these commands is not particularly user friendly. The mitmproxy web interface can help, which is accessed at http://127.0.0.1:8081 by default. In the web GUI, mitmproxy shows the HTTP command exchange history. To install the required certificate on the client system, go to File | Install Certificates .

Figure 2: You can follow the client's communication on the console and interactively intervene in the data exchange.

The Options tab offers various customization options for the web interface that mainly relate to the display, but you can also work with block lists and add an empty response for specific requests. Mitmproxy supports the following modes in addition to the Regular default mode:

• Transparent
• WireGuard
• Reverse Proxy
• Upstream Proxy
• SOCKS Proxy
• DNS Server

The choice of mode depends on whether you want to monitor the traffic from a client or a single server. If you want to examine the traffic reaching your own web server from the Internet, you need to enable the Reverse Proxy variant. In this case, mitmproxy acts like a "normal" server that fields the requests from the Internet or from an analysis client and passes them on to the target system.

To make it easier to find the goodies, switch to the Start tab and enter a search term, which is a good way of restricting the view to specific sources, services, or file types. The good thing about this is that if the web GUI is open and you access the server in a second browser window, you can trace the commands in mitmproxy in real time.

The individual requests and responses can be examined both at the console level and in the web interface, but the web interface is far more convenient. You can play back the logged exchanges or select Edit to edit requests and forward them to the target server after doing so. It is easy to apply filters at the console level, intercept the traffic, and save the markup locally.

In view of the huge volumes of data that mitmproxy delivers, it is not always easy to keep track of the content in which you are interested, which is where the Highlight function comes in handy. Clicking on the Highlight button opens a selection menu where you can highlight specific content. You can limit the selection to methods or content types or even use a regular expression.

Advanced Analysis Techniques

One of mitmproxy's special features is the Interception function, which lets you intercept and manipulate requests. Intercepting is not normally desirable because it impairs the browsing experience. To keep you from taking down the entire data exchange, mitmproxy uses a different approach and intercepts requests selectively. You can set this up in the web GUI or at the console with the ~u <regex> flow filter, and you can use the ~q option to avoid intercepting the responses. An ampersand (&) lets you combine multiple filter options.

Once you have interrupted a request, the next step is to modify it to suit your needs. If you are working with the web GUI, first enable Edit mode and select the entry with the request that you want to edit (e.g., you could manipulate the Path or user-agent option).

If you want to make the changes at the console level, press E to enable Edit mode; in the web GUI, open the Request tab (Figure 3) and navigate to the entry in which you are interested. Make your changes and exit Edit mode. To resume the interrupted data flow, press the A button at the console or click Resume in the GUI.

Figure 3: The web interface makes analyzing and manipulating web traffic easy.

Another special feature of mitmproxy is that you can replay previous data flows repeatedly. The tool supports two types of replay requests:

• Client-side replay retransmits previous client requests to the server.
• Server-side replay replays the server's responses to previously recorded requests.

You can use both variants at the console and in the web GUI. To begin, select the desired requests with filter expressions. To repeat these, press the r key at the console or click Replay in the GUI. You can modify the request here, too.

Thanks to its modular architecture, you can extend mitmproxy's functionality and draw on the development work of an active community. Table 1 summarizes the most interesting extensions. For example, the log-events.py script runs at console level and generates warnings.

Excluding Domains

When analyzing traffic, it might make sense to ignore certain domains – for two main reasons. First, some of the traffic might be protected by certificate pinning, and second, some of the traffic might simply be beyond your scope of interest.

In the first case, for example, Windows updates or access to Apple's App Store will not work if you have set up mitmproxy in the middle. If some of the content is not relevant to you, simply hide it with the view_filter option.

To exclude hosts from monitoring, use the ignore_hosts option and specify this with a regular expression in the host:port string format. The command line alias is --ignore-hosts regex, and the complete command is:

mitmproxy -ignore-hosts '^example\.de:443$'

If you want to restrict recording to specific domains, use the --allow-hosts option.

Another great feature of mitmproxy is that the environment has a modular architecture, and you extend the feature set with the help of Python scripts. A major share of the existing feature set is already based on integrated add-ons. The structure and design of these scripts are kept so simple that it is easy to adapt them or develop your own scripts. Almost 30 scripts are currently available for download from the GitHub project site [5].