On 2025-02-03 14:40, Ron / BCLUG via talk wrote:
Alvin Starr via talk wrote on 2025-02-03 10:20:
Think of it like this:
$ python prog1.py &
$ python prog2.py &
If my memory serves me correctly all the code is shared between the processes Its just the data which is private to each process. Now in the case of an interpreter, the xxx.py programs are just data from the OS point of view so that is duplicated.
https://www.kernel.org/doc/gorman/html/understand/understand007.html looks to describe the overall memory management of Linux.
I'd like to read more about that if anyone has links that indicate launching multiple programs (i.e. Python scripts) only load python once.
A way to check this would be to load a single python interpreter and look at the memory used. Then load multiple copies of the same program and see if the memory usage is N*mem_used and not something less than N*mem_used.
The systemctl output shows multiple pythons running, each with their own PID (lines chopped to avoid wrapping):
Tasks: 9 (limit: 4915) Memory: 95.2M CGroup: /system.slice/mailman.service ├─25091 /usr/bin/python /usr/lib/mailman/bin/mailmanctl ├─25092 /usr/bin/python /var/lib/mailman/bin/qrunner ├─25093 /usr/bin/python /var/lib/mailman/bin/qrunner ├─25094 /usr/bin/python /var/lib/mailman/bin/qrunner ├─25095 /usr/bin/python /var/lib/mailman/bin/qrunner ├─25096 /usr/bin/python /var/lib/mailman/bin/qrunner ├─25097 /usr/bin/python /var/lib/mailman/bin/qrunner ├─25098 /usr/bin/python /var/lib/mailman/bin/qrunner └─25099 /usr/bin/python /var/lib/mailman/bin/qrunner
Surely that means python is loaded and re-loaded?
you have other things going on here. There is the memory used up by the interpreter reading in the various bits of python.py and python.pyc files Then you have the memory used by the actual execution of the python programs.
I'm going to dig around a bit and see if I can account for that 95.2M RAM usage...
Python2.7 interpreter size:
# ls -l $(which python2.7) -h ... 3.7M 2022-02-06 15:16 /usr/bin/python2.7*
The qrunner which is a python script that is invoked repeatedly with different options (which are, in turn other scripts):
# ls -l /usr/lib/mailman/bin/qrunner ... 9617 2022-06-08 14:24 /usr/lib/mailman/bin/qrunner*
And these are the scripts loaded by qrunner:
ll /usr/lib/mailman/Mailman/Queue/ArchRunner.py* ... 3099 2022-06-08 14:24 /usr/.../Mailman/Queue/ArchRunner.py ... 1831 2022-06-22 17:55 /usr/.../Mailman/Queue/ArchRunner.pyc
# ll /usr/lib/mailman/Mailman/Queue/BounceRunner.py* ... 15901 2022-06-08 14:24 /usr/.../Mailman/Queue/BounceRunner.py ... 8830 2022-06-22 17:55 /usr/.../Mailman/Queue/BounceRunner.pyc
# ll /usr/lib/mailman/Mailman/Queue/CommandRunner.py* ... 11882 2022-06-08 14:24 /usr/.../Mailman/Queue/CommandRunner.py ... 7639 2022-06-22 17:55 /usr/.../Mailman/Queue/CommandRunner.pyc
# ll /usr/lib/mailman/Mailman/Queue/IncomingRunner.py* ... 9080 2022-06-08 14:24 /usr/.../Mailman/Queue/IncomingRunner.py ... 2632 2022-06-22 17:55 /usr/.../Mailman/Queue/IncomingRunner.pyc
# ll /usr/lib/mailman/Mailman/Queue/NewsRunner.py* ... 7365 2022-06-08 14:24 /usr/.../Mailman/Queue/NewsRunner.py ... 4303 2022-06-22 17:55 /usr/.../Mailman/Queue/NewsRunner.pyc
# ll /usr/lib/mailman/Mailman/Queue/OutgoingRunner.py* ... 5707 2022-06-08 14:24 /usr/.../Mailman/Queue/OutgoingRunner.py ... 3282 2022-06-22 17:55 /usr/.../Mailman/Queue/OutgoingRunner.pyc
# ll /usr/lib/mailman/Mailman/Queue/VirginRunner.py* ... 1732 2022-06-08 14:24 /usr/.../Mailman/Queue/VirginRunner.py ... 1311 2022-06-22 17:55 /usr/.../Mailman/Queue/VirginRunner.pyc
# ll /usr/lib/mailman/Mailman/Queue/RetryRunner.py* ... 1481 2022-06-08 14:24 /usr/.../Mailman/Queue/RetryRunner.py ... 1439 2022-06-22 17:55 /usr/.../Mailman/Queue/RetryRunner.pyc
Adding the *.pyc file sizes:
1831 + 8830 + 7639 + 2632 + 4303 + 3282 + 1311 + 1439 = 31267 bytes.
Each of those is invoked via qrunner (8 instances at 9617 bytes):
9617 * 8 = 76936 bytes
So far, I can account for 76936 + 31267 bytes = 108203 bytes.
A couple of things. The .py and pyc files are only the python code portion and they may load in other things like files from the disk. They also likely reference other python libraries that reference other python libraries ..... So you need to know all the python code that is loaded up when you start your program. You also have things that are staticlly allocated memory within the python program that may be a few tens of bites of code but could be arbitrary large memory blocks.
Need this process too:
# ls -l /usr/lib/mailman/bin/mailmanctl ... 21431 2022-06-08 14:24 /usr/lib/mailman/bin/mailmanctl*
Up to 21431 + 108203 = 129634 bytes.
# systemctl status mailman Tasks: 9 (limit: 4915) Memory: 95.2M
My numbers are way, *way* short of 96MB so let's add in some Python2.7 interpreters (one for each qrunner) as qrunner is a python script, and one for mailmanctl:
# file /var/lib/mailman/bin/qrunner /var/lib/mailman/bin/qrunner: a /usr/bin/python script, ASCII text executable
9 * 3775416 = 33,978,744 MB
So, accounting for all the compiled python code plus the interpreters gets to about ⅓ the RAM usage of Mailman v2.
That seems... reasonable?
Have I just wasted a bunch of time and entirely gotten the part about multiple Python interpreters loaded in memory? --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
-- Alvin Starr || land: (647)478-6285 Netvel Inc. || home: (905)513-7688 alvin@netvel.net ||