Mitmproxy GitHub: Understanding the Memory Usage Growth Pattern

When running mitmdump, a tool used for debugging and testing web applications, users may notice a peculiar pattern of memory usage growth. This article aims to explain this phenomenon and provide potential solutions.

Firstly, it's essential to ignore the cycles of memory allocation and deallocation and focus on understanding the overall trend. To do so, import necessary libraries such as `collections`, `gc`, `os`, and `signal` from the `mitmproxy` module. Next, define a function `debug()` that increments a counter for each request processed by mitmdump.

To investigate this issue further, run mitmdump with debugging enabled (`-set proxy_debug -vvv`) but this may not affect the behavior. The problem appears to have been present since the introduction of the current Git master version of mitmproxy.

To reproduce the memory usage growth pattern, try running mitmdump for an extended period while using multiple browsers with a large number of tabs open. This will simulate real-world scenarios where multiple requests are processed concurrently. Rarely, mitmdump may experience larger memory allocations and deallocations, but this is expected behavior as it handles requests and frees memory after processing each request.

Despite the presence of background traffic, the likelihood of zero traffic is low due to websites constantly loading content in the background. Additionally, running a script that terminates mitmdump if it exceeds 3GB can help prevent system crashes.

The provided graph shows mitmdump's manual termination due to debugging and daily restarts at midnight. The growth rate typically ranges from 500MB to 1GB per hour, which is higher than the rate of data dumped (under 1GB per day).

Mitmdump's memory usage appears as a sawtooth pattern, rising until it is terminated. This behavior is unexpected, given that mitmdump should not keep flows in memory after writing them out.

To summarize, mitmproxy running interactively displays all data in the UI, which is reasonable; however, mitmdump should have no reason to keep flows in memory after writing them out. The constant growth of memory usage may be attributed to allocating memory for request/response data and never freeing it.

Mitmproxy's GitHub issue #4451 discusses an unrelated problem, but the described behavior warrants a separate ticket. Running mitmdump continuously with multiple browsers on multiple computers may lead to unexpected memory usage patterns. By understanding this phenomenon, developers can better troubleshoot and optimize their use of mitmdump.

Mitmproxy github