Memory profiling forms a very important aspect of the performance analysis of an application. When building an application, there are places where we may implement an incorrect mechanism of dealing with the dynamically allocated objects and hence may land up in a situation where these objects which are no longer in use are still having a reference pointing to them preventing their garbage collection by the Garbage Collector.

This results in the growth of the application-memory usage over time, causing the application to come to a halt once the system runs out of memory that can be allocated to the application for performing its regular activities.

Now, to address these kinds of issues, we don't require a profiler that will help us analyze the call stack of the application and provide us details about how much time an individual call took. Instead, what we need here is a profiler that can tell us about the memory trends of an application, such as how much memory individual methods might be consuming and how that memory grows as the application continues to run.

This is the place where memory_profiler comes in, which is a third-party module that we can easily include in our application to allow memory profiling. But, before we dive into how to use memory_profiler, we need to get the module into our development environment first. The following line of code fetches the required module into our development environment:

pip install memory_profiler

Once the memory profiler has been fetched into the developer environment, we are now ready to get up and running with it. Let's take a look at a sample program and see how we can use memory_profiler to understand the memory usage patterns of our application.

The following code snippet shows us an example of how we can use memory_profiler:

from memory_profiler import profile

@profile
def calc_sum():
    sum = 0
    for i in range(100):
        sum = sum + i
    print(str(sum))

if __name__ == '__main__':
    calc_sum()

Now, with the code in place, let's try to understand what we did here.

At the start, we imported a decorator known as profile, which is provided by the memory_profiler library. This decorator is used to notify memory_profiler of which methods needs to be profiled for the memory usage.

To enable memory profiling for a method, all we need to do is decorate that method with the decorator. For example, in our sample application code, we decorated the calc_sum() method with the decorator.

Now, let's run our sample code and see what we get as an output by running the following command:

python3 memory_profile_example.py

Once the command is executed, we get the following output:

4950
Filename: memory_profile.py

Line # Mem usage Increment Line Contents
================================================
     3 11.6 MiB 11.6 MiB @profile
     4 def calc_sum():
     5 11.6 MiB 0.0 MiB sum = 0
     6 11.6 MiB 0.0 MiB for i in range(100):
     7 11.6 MiB 0.0 MiB sum = sum + i
     8 11.6 MiB 0.0 MiB print(str(sum))

As we can see from the preceding output, we got some detailed statistics about the memory allocation for the method. The output provided us information about how much memory was being used and how much memory increment each of the steps caused to the application.

Now, let's take one more example to see how the memory allocation changes when one method calls another method. The following code showcases this:

from memory_profiler import profile

@profile
def calc_sum():
    sum = 0
    for i in range(100):
        sum = sum + i
    say_hello()
    print(str(sum))

def say_hello():
    lst = []
    for i in range(10000):
        lst.append(i)

if __name__ == '__main__':
    calc_sum()

On executing the preceding code, we get to see the following output:

Line # Mem usage Increment Line Contents
================================================
     3 11.6 MiB 11.6 MiB @profile
     4 def calc_sum():
     5 11.6 MiB 0.0 MiB sum = 0
     6 11.6 MiB 0.0 MiB for i in range(100):
     7 11.6 MiB 0.0 MiB sum = sum + i
     8 11.7 MiB 0.1 MiB say_hello()
     9 11.7 MiB 0.0 MiB print(str(sum))

As we can see, when a call to the say_hello() method was made, the call caused an increment of 0.1 MB of memory usage. This is quite a handy library in case we suspect that there's some memory leak that may be happening somewhere in the code.