Develop Reference

How to track a memory-usage growing object in a python script

I have a python script, whose memory usage (observed in top) is growing indefinitely when the script is running.
My script is not supposed to store anything. It just receives some request (REST API), processes the request, and returns some result. The expected behaviour is that the memory usage stays constant.
I'm looking for a suitable memory profiler tool that would allow me to identify where the growing object is located in the code.
I had a look at FIL-profiler, but it seems to aim at considering "peak memory usage", which is I believe not my case.
thanks
EDIT1: I wanted to give pyrasite a try, but I could not make it work. Since the last update of this project is 2012 May, it may be that it is not anymore compatible with the current python (I'm using 3.8)
EDIT2: I gave a try to Pympler, following this post. I added the following code to my script:
from pympler import muppy, summary
all_objects = muppy.get_objects()
sum = summary.summarize(all_objects)
summary.print_(sum)
This outputs:
types | # objects | total size
========= | =========== | ============
dict | 7628 | 3.15 MB
str | 24185 | 2.71 MB
type | 1649 | 1.39 MB
tuple | 13722 | 1.35 MB
code | 7701 | 1.06 MB
set | 398 | 453.06 KB
....
Which does not show any suspiciously large object (the scripts, when running, accumulates a memory usage going to GB scale)
EDIT3:
I tried with tracemalloc: I put a tracemalloc.start() at the beginning of the script, then, at the end of the script, but before stopping (so that the memory usage is visibly very high, according to top), I do a
snapshot = tracemalloc.take_snapshot()
display_top(snapshot)
This displays the lines of code with maximal memory usage. But still, nothing in comparison to that observed in top.
I tried also a gc.collect(), but this has no effect.

I could not take benefit of any of the proposed memory profiler. It turned out that the problem lies in a possible bug in the library ujson, which is written in C.
I guess that this is the reason why all those python memory profiler could not help here.
I think I have to close the issue. A remaining question would be: Is there any python tool that allow to track a memory problem happening in a C module?

Assuming you are using the "VIRT" column from top, you cannot extrapolate from that number to an assumption that the number of your python objects are growing, or at least growing enough to explain the total size of the virtual address space.
For example, did you know that python uses pthreads underneath its threading code? That is significant because pthreads takes, as the default stack size, the value from "ulmimit -s" * 1K. So the default stack size for any new threads in python is often 8MB or even 40MB on some variants of Linux, unless you explicitly change the "ulimit -s" value in the parent of your process. Many server applications are multi-threaded so even having 2 additional threads would result in more virtual memory than you are showing from the Pympler output. You have to know how many threads you have in your process and what is the default stack size to understand how threads might contribute to the total VIRT value.
Also, underneath its own allocation mechanism python is using a mixture of mmap and malloc. In the case of malloc, if the program is multi-threaded, on linux libc malloc will use multiple arenas, which reserve 64MB ranges (called heaps) at a time, but only make the starts of these heaps readable and writable and leave the tails of these ranges inaccessible until the memory is needed. Those inaccessible tails are often large but don't really matter at all in terms of the committed memory for the process because inaccessible pages don't require any physical memory or swap space. Nonetheless, top counts in "VIRT" the entire range, including both the accessible start at the beginning of the range and the inaccessible start at the end of the range.
For example, consider the rather tiny python program, in which the main thtead starts 16 additional threads, each of which don't use much memory in python allocations:
import threading
def spin(seed):
l = [ i * seed for i in range(64) ]
while True:
l = [ i * i % 97 for i in l ]
for i in range(16):
t = threading.Thread(target=spin, args=[i])
t.start()
We would not expect that program to result in all that large a process but here is what we see under top, looking at just that one process, which shows way more than 1GB of VIRT:
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.0%us, 1.9%sy, 3.3%ni, 93.3%id, 0.5%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 264401648k total, 250450412k used, 13951236k free, 1326116k buffers
Swap: 69205496k total, 17321340k used, 51884156k free, 104421676k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9144 tim 20 0 1787m 105m 99m S 101.8 0.0 13:40.49 python
We can understand the high VIRT value for such a program (and your program for that matter) by taking a live core of the running program (using gcore, for example) and opening the resulting core using chap, open source which can be found at https://github.com/vmware/chap and running commands as shown:
chap> summarize writable
17 ranges take 0x2801c000 bytes for use: stack
14 ranges take 0x380000 bytes for use: python arena
16 ranges take 0x210000 bytes for use: libc malloc heap
1 ranges take 0x1a5000 bytes for use: libc malloc main arena pages
11 ranges take 0xa9000 bytes for use: used by module
1 ranges take 0x31000 bytes for use: libc malloc mmapped allocation
2 ranges take 0x7000 bytes for use: unknown
62 writable ranges use 0x28832000 (679,682,048) bytes.
From the above we can see that the biggest single use of memory is 17 stacks. If we were to use "describe writable" we would see that 16 of those stacks take 40MB each, and this is all because we have neglected to explicitly tune the stack size to something more reasonable.
We can do something similar with inaccessible (not readable, writable or executable) regions and see that 16 "libc malloc heap tail reservation" ranges take almost 1GB of VIRT. This doesn't as it turns out, really matter for the committed memory of the process but it does make a rather scary contribution to the VIRT number.
chap> summarize inaccessible
16 ranges take 0x3fdf0000 bytes for use: libc malloc heap tail reservation
11 ranges take 0x15f9000 bytes for use: module alignment gap
16 ranges take 0x10000 bytes for use: stack overflow guard
43 inaccessible ranges use 0x413f9000 (1,094,684,672) bytes.
The reason that there are 16 such ranges is that each of the spinning threads was repeatedly making allocations, and this caused libc malloc, running behind the scene because it gets used by the python allocator, to carve out 16 arenas.
You could do a similar command for read only memory ("summarize readonly") or you could get more detail for any of the commands by changing "summarize" to "describe".
I can't say exactly what you will find when you run this on your own server, but it seems quite likely for a REST server to be multi-threaded, so I would guess that this will be a non-trivial contribution to the number TOP is showing you.
This still doesn't answer why the number is continuing to climb, but if you look at those numbers you can see where to look next. For example, in the above summary allocations that are handled by python without resorting to mmap don't take that much memory at only 3.5MB:
14 ranges take 0x380000 bytes for use: python arena
In your case the number might be larger, so you might try any of the following:
describe used
describe free
summarize used
summarize free
Note that the above commands will also cover native allocations (for example ones done by a shared library) as well as python allocations.
Following similar steps on your own program should give you a much better understanding of those "top" numbers you are seeing.

Understanding memory growth of a Python process (VmRSS vs gc.get_objects)

Without going into algorithmic details, lets just say that my code sequentially processes a list of inputs:
inputs = [2,5,6,7,8,10,12,13,14,15,16,17,18,19,20,21]
for i in inputs:
process_input(i)
For simplicity, lets consider process_input to be a state-less black-box.
I know that this site is full of questions about finding memory leaks in Python code, but this is not what this question is about. Instead, I'm trying to understand the memory consumption of my code over time and whether it might suffer from leaking memory.
In particular, I'm trying to understand a discrepancy of two distinct indicators of memory usage:
The number of allocated objects (reported by gc.get_objects) and
the actually used amount of physical memory (read from VmRSS on a Linux system).
To study these two indicators, I expanded the original code from above as follows:
import time, gc
def get_current_memory_usage():
with open('/proc/self/status') as f:
memusage = f.read().split('VmRSS:')[1].split('\n')[0][:-3]
return int(memusage.strip()) / (1024 ** 2)
inputs = [2,5,6,7,8,10,12,13,14,15,16,17,18,19,20,21]
gc.collect()
last_object_count = len(gc.get_objects())
for i in inputs:
print(f'\nProcessing input {i}...')
process_input(i)
gc.collect()
time.sleep(1)
memory_usage = get_current_memory_usage()
object_count = len(gc.get_objects())
print(f'Memory usage: {memory_usage:.2f} GiB')
print(f'Object count: {object_count - last_object_count:+}')
last_object_count = object_count
Note that process_input is state-less, i.e. the order of the inputs does not matter. Thus, we would expect both indicators to be about the same before running process_input and afterwards, right? Indeed, this is what I observe for the number of allocated objects. However, the consumption of memory grows steadily:
Now my core question: Do these observations indicate a memory leak? To my understanding, memory leaking in Python would be indicated by a growth of allocated objects, which we do not observe here. On the other hand, why does the memory consumption grow steadily?
For further investigation, I also ran a second test. For this test, I repeatedly invoked process_input(i) using a fixed input i (five times each) and recorded the memory consumption in between of the iterations:
For i=12, the memory consumption remained constant at 10.91 GiB.
For i=14, the memory consumption remained constant at 7.00 GiB.
I think, these observations make the presence of a memory leak even more unlikely, right? But then, what could be a possible explanation for why the memory consumption is not falling in between of the iterations, given that process_input is state-less?
The system has 32 GiB RAM in total and is running Ubuntu 20.04. Python version is 3.6.10. The process_input function uses several third-party libraries.

In general RSS is not a particularly good indicator because it is "resident" set size and even a rather piggish process, in terms of committed memory, can have a modest RSS as memory can be swapped out. You can look at /proc/self/smaps and add up the size of the writable regions to get a much better benchmark.
On the other hand, if there is actually growth, and you want to understand why, you need to look at the actual dynamically allocated memory. What I'd suggest for this is using https://github.com/vmware/chap
To do this, just make that 1 second sleep a bit longer, put a print just before the call to sleep, and use gcore from another session to gather a live core during a few of those sleeps.
So lets say you have cores gathered from when the input was 14 and when it was 21. Look at each of the cores using chap, for example, with the following commands:
count used
That will give you a good view of allocations that have been requested but not released. If the numbers are much larger for the later core, you probably have some kind of growth issue. If those numbers do differ by quite a lot, use
summarize used
If you have growth, it is possible that there is a leak (as opposed to some container simply expanding). To check this, you can try commands like
count leaked
show leaked
From there you should probably look at the documentation, depending on what you find.
OTOH if used allocations are not the issue, maybe try the following, to see memory for allocations that have been released but are part of larger regions of memory that cannot be given back to the operating system because parts of those regions are still in use:
count free
summarize free
If neither "used" allocations or "free" allocations are the issue, you might try:
summarize writable
That is a very high level view of all writable memory. For example, you can see things like stack usage...

is there any alternative to sys.getsizeof() in PyPy?

I am trying to run a Python (2.7) script with PyPy but I have encountered the following error:
TypeError: sys.getsizeof() is not implemented on PyPy.
A memory profiler using this function is most likely to give results
inconsistent with reality on PyPy. It would be possible to have
sys.getsizeof() return a number (with enough work), but that may or
may not represent how much memory the object uses. It doesn't even
make really sense to ask how much *one* object uses, in isolation
with the rest of the system. For example, instances have maps,
which are often shared across many instances; in this case the maps
would probably be ignored by an implementation of sys.getsizeof(),
but their overhead is important in some cases if they are many
instances with unique maps. Conversely, equal strings may share
their internal string data even if they are different objects---or
empty containers may share parts of their internals as long as they
are empty. Even stranger, some lists create objects as you read
them; if you try to estimate the size in memory of range(10**6) as
the sum of all items' size, that operation will by itself create one
million integer objects that never existed in the first place.
Now, I really need to check the size of one nested dict during the execution of the program, is there any alternative to sys.getsizeof() I can use in PyPy? If not, how would I check for the size of a nested object in PyPy?

Alternatively you can gauge the memory usage of your process using
import resource
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
As your program is executing, getrusage will give total memory consumption of the process in number of bytes or kilobytes. Using this information you can estimate the size of your data structures, and if you begin to use say 50% of your machine's total memory, then you can do something to handle it.

64 bit Python causes freeze

Being new to programming, I was playing around with Python with this script:
import itertools
Lists=list(itertools.permutations(['a','b','c','d','e','f','g','h','i','j','k']))
print(len(Lists))
On 32-bit Python it will cause a memory overflow error. However when trying it on 64-bit Python and watching Task Manager, Python uses 4 GB of memory (I have 8 GB of RAM) then my computer freezes, and I have to restart it.
Is this normal behavior? How can I fix this, or limit how much memory Python has access to?
Also if I converted something like this to a .exe file (used this script as testing for something else) would it freeze other computers?

The function itertools.permutations() returns a generator that lazily computes all possible permutations of the given sequence in lexicographical order. Then your code stores all these permutations explicitly in a list.
Your sequence contains 11 letters. For your input, there are 11! = 39 916 800 permutations. Python is not particularly memory-efficient; for each of the 40 million permutations, these values need to be stored:
A pointer to the list object. (8 bytes on 64-bit Python)
The list container itself, containing 11 pointers to strings and slack space. (8× 8 bytes)
Thus at least 96 bytes are used per permutation. Adding some padding and miscellaneous waste, we can estimate that each permutation uses 150 bytes of memory. Multiply this by 40 million, and we get 6 gigabytes.
This high memory usage explains why your program dies on 32-bit Python (cannot use more than 4 GB of RAM, and in practice is limited by 2 GB). Also, when the process consumes a lot of memory, it can cause thrashing in the page/swap file if one is enabled.
One way to limit Python's memory limit is through mechanisms provided through the operating system, such as ulimit. Another way is by consulting the resource module.

Can we use a python variable to hold an entire file?

Provided that we know that all the file will be loaded in memory and we can afford it,
what are the drawbacks (if any) or limitations (if any) of loading an entire file (possibly a binary file) in a python variable. If this is technically possible, should this be avoided, and why ?
Regarding file size concerns, to what maximum size this solution should be limited ?. And why ?
The actual loading code could be the one proposed in this stackoverflow entry.
Sample code is:
def file_get_contents(filename):
with open(filename) as f:
return f.read()
content = file_get_contents('/bin/kill')
... code manipulating 'content' ...
[EDIT]
Code manipulation that comes to mind (but is maybe not applicable) is standard list/strings operators (square brackets, '+' signs) or some string operators ('len', 'in' operator, 'count', 'endswith'/'startswith', 'split', 'translation' ...).

Yes, you can
The only drawback is memory usage, and possible also speed if the file is big.
File size should be limited to how much space you have in memory.
In general, there are better ways to do it, but for one-off scripts where you know memory is not an issue, sure.

While you've gotten good responses, it seems nobody has answered this part of your question (as often happens when you ask many questions in a question;-)...:
Regarding file size concerns, to what
maximum size this solution should be
limited ?. And why ?
The most important thing is, how much physical RAM can this specific Python process actually use (what's known as a "working set"), without unduly penalizing other aspects of the overall system's performance. If you exceed physical RAM for your "working set", you'll be paginating and swapping in and out to disk, and your performance can rapidly degrade (up to a state known as "thrashing" were basically all available cycles are going to the tasks of getting pages in and out, and negligible amounts of actual work can actually get done).
Out of that total, a reasonably modest amount (say a few MB at most, in general) are probably going to be taken up by executable code (Python's own executable files, DLLs or .so's) and bytecode and general support datastructures that are actively needed in memory; on a typical modern machine that's not doing other important or urgent tasks, you can almost ignore this overhead compared to the gigabytes of RAM that you have available overall (though the situation might be different on embedded systems, etc).
All the rest is available for your data -- which includes this file you're reading into memory, as well as any other significant data structures. "Modifications" of the file's data can typically take (transiently) twice as much memory as the file's contents' size (if you're holding it in a string) -- more, of course, if you're keeping a copy of the old data as well as making new modified copies/versions.
So for "read-only" use on a typical modern 32-bit machine with, say, 2GB of RAM overall, reading into memory (say) 1.5 GB should be no problem; but it will have to be substantially less than 1 GB if you're doing "modifications" (and even less if you have other significant data structures in memory!). Of course, on a dedicated server with a 64-bit build of Python, a 64-bit OS, and 16 GB of RAM, the practical limits before very different -- roughly in proportion to the vastly different amount of available RAM in fact.
For example, the King James' Bible text as downloadable here (unzipped) is about 4.4 MB; so, in a machine with 2 GB of RAM, you could keep about 400 slightly modified copies of it in memory (if nothing else is requesting memory), but, in a machine with 16 (available and addressable) GB of RAM, you could keep well over 3000 such copies.

with open(filename) as f:
This only works on Python 2.x on Unix. It won't do what you expect on Python 3.x or on Windows, as these both draw a strong distinction between text and binary files. It's better to specify that the file is binary, like this:
with open(filename, 'rb') as f:
This will turn off the OS's CR/LF conversion on Windows, and will force Python 3.x to return a byte array rather than Unicode characters.
As for the rest of your question, I agree with Lennart Regebro's (unedited) answer.

The sole issue you can run into is memory consumption: Strings in Python are immutable. So when you need to change a byte, you need to copy the old string:
new = old[0:pos] + newByte + old[pos+1:]
This needs up to three times the memory of old.
Instead of a string, you can use an array. These offer much better performance if you need to modify the contents and you can create them easily from a string.

You can also use Python's v3 feature:
>>> ''.join(open('htdocs/config.php', 'r').readlines())
"This is the first line of the file.\nSecond line of the file"
Read more here http://docs.python.org/py3k/tutorial/inputoutput.html

Yes you can -provided the file is small enough-.
It is even very pythonic to further convert the return from read() to any container/iterable type as with say, string.split(), along with associated functional programming features to continue treating the file "at once".