Develop Reference

Shared-memory IPC solution for both Linux and Windows

Below is a simple and perfect solution on Windows for IPC with shared memory, without having to use networking / sockets (that have annoying limits on Windows).
The only problem is that it's not portable on Linux:
Avoiding the use of the tag parameter will assist in keeping your code portable between Unix and Windows.
Question: is there a simple way built-in in Python, without having a conditional branch "if platform is Windows, if platform is Linux" to have a shared-memory mmap?
Something like
mm = sharedmemory(size=2_000_000_000, name="id1234") # 2 GB, id1234 is a global
# id available for all processes
mm.seek(1_000_000)
mm.write(b"hello")
that would internally default to mmap.mmap(..., tagname="id1234") on Windows and use /dev/shm on Linux (or maybe even a better solution that I don't know?), and probably something else on Mac, but without having to handle this manually for each different OS.
Working Windows-only solution:
#server
import mmap, time
mm = mmap.mmap(-1, 1_000_000_000, tagname="foo")
while True:
mm.seek(500_000_000)
mm.write(str(time.time()).encode())
mm.flush()
time.sleep(1)
# client
import mmap, time
mm = mmap.mmap(-1, 1_000_000_000, tagname="foo")
while True:
mm.seek(500_000_000)
print(mm.read(128))
time.sleep(1)

The easiest way is to use python with version >=3.8, it has added a built-in abstraction for shared memory,
it works on both windows and linux
https://docs.python.org/3.10/library/multiprocessing.shared_memory.html
The code will look something like this:
Process #1:
from multiprocessing import shared_memory
# create=true to create a new shared memory instance, if it already exists with the same name, an exception is thrown
shm_a = shared_memory.SharedMemory(name="example", create=True, size=10)
shm_a.buf[:3] = bytearray([1, 2, 3])
while True:
do_smt()
shm_a.close()
Process #2:
from multiprocessing import shared_memory
# create=false, use existing
shm_a = shared_memory.SharedMemory(name="example", size=10)
print(bytes(shm.buf[:3]))
# [0x01, 0x02, 0x03]
while True:
do_smt()
shm_a.close()
Otherwise, I think there are no common good solutions and you will need to reinvent the wheel :)

Personally this has worked well for me
Option 1: http://www.inspirel.com/yami4/
The YAMI4 suite for general computing is a multi-language and multi-platform package.
Several Operating systems:
Sample code
Microsoft Windows, POSIX (Linux, Max OS X, FreeBSD, ...), QNX (with native IPC messaging), FreeRTOS, ThreadX, TI-RTOS. Programming languages: C++, Ada, Java, .NET, Python, Wolfram.
Option 2: ZeroMq https://zeromq.org/

Pygraphviz crashes after drawing 170 graphs

I am using pygraphviz to create a large number of graphs for different configurations of data. I have found that no matter what information is put in the graph the program will crash after drawing the 170th graph. There are no error messages generated the program just stops. Is there something that needs to be reset if drawing this many graphs?
I am running Python 3.7 on a Windows 10 machine, Pygraphviz 1.5, and graphviz 2.38
for graph_number in range(200):
config_graph = pygraphviz.AGraph(strict=False, directed=False, compound=True, ranksep='0.2', nodesep='0.2')
# Create Directory
if not os.path.exists('Graph'):
os.makedirs('Graph')
# Draw Graph
print('draw_' + str(graph_number))
config_graph.layout(prog = 'dot')
config_graph.draw('Graph/'+str(graph_number)+'.png')

I was able to constantly reproduce the behavior with:
Python 3.7.6 (pc064 (64bit), then also with pc032)
PyGraphviz 1.5 (that I built - available for download at [GitHub]: CristiFati/Prebuilt-Binaries - Various software built on various platforms. (under PyGraphviz, naturally).
Might also want to check [SO]: Installing pygraphviz on Windows 10 64-bit, Python 3.6 (#CristiFati's answer))
Graphviz 2.42.2 ((pc032) same as #2.)
I suspected an Undefined Behavior somewhere in the code, even if the behavior was precisely the same:
OK for 169 graphs
Crash for 170
Did some debugging (added some print(f) statements in agraph.py, and cgraph.dll (write.c)).
PyGraphviz invokes Graphviz's tools (.exes) for many operations. For that, it uses subprocess.Popen and communicates with the child process via its 3 available streams (stdin, stdout, stderr).
From the beginning I noticed that 170 * 3 = 510 (awfully close to 512 (0x200)), but didn't pay as much attention as I should have until later (mostly because the Python process (running the code below) had no more than ~150 open handles in Task Manager (TM) and also Process Explorer (PE)).
However, a bit of Googleing revealed:
[SO]: Is there a limit on number of open files in Windows (#stackprogrammer's answer) (and from here)
[MS.Learn]: _setmaxstdio (which states (emphasis is mine)):
C run-time I/O now supports up to 8,192 files open simultaneously at the low I/O level. This level includes files opened and accessed using the _open, _read, and _write family of I/O functions. By default, up to 512 files can be open simultaneously at the stream I/O level. This level includes files opened and accessed using the fopen, fgetc, and fputc family of functions. The limit of 512 open files at the stream I/O level can be increased to a maximum of 8,192 by use of the _setmaxstdio function.
[SO]: Python: Which command increases the number of open files on Windows? (#NorthCat's answer)
Below is your code that I modified for debugging and reproducing the error. It needs (for code shortness' sake, as same thing can be achieved via CTypes) the PyWin32 package (python -m pip install pywin32).
code00.py:
#!/usr/bin/env python
import os
import sys
#import time
import pygraphviz as pgv
import win32file as wfile
def handle_graph(idx, dir_name):
graph_name = "draw_{:03d}".format(idx)
graph_args = {
"name": graph_name,
"strict": False,
"directed": False,
"compound": True,
"ranksep": "0.2",
"nodesep": "0.2",
}
graph = pgv.AGraph(**graph_args)
# Draw Graph
img_base_name = graph_name + ".png"
print(" {:s}".format(img_base_name))
graph.layout(prog="dot")
img_full_name = os.path.join(dir_name, img_base_name)
graph.draw(img_full_name)
graph.close() # !!! Has NO (visible) effect, but I think it should be called anyway !!!
def main(*argv):
print("OLD max open files: {:d}".format(wfile._getmaxstdio()))
# 513 is enough for your original code (170 graphs), but you can set it up to 8192
#wfile._setmaxstdio(513) # !!! COMMENT this line to reproduce the crash !!!
print("NEW max open files: {:d}".format(wfile._getmaxstdio()))
dir_name = "Graph"
# Create Directory
if not os.path.isdir(dir_name):
os.makedirs(dir_name)
#ts_global_start = time.time()
start = 0
count = 170
#count = 1
step_sleep = 0.05
for i in range(start, start + count):
#ts_local_start = time.time()
handle_graph(i, dir_name)
#print(" Time: {:.3f}".format(time.time() - ts_local_start))
#time.sleep(step_sleep)
handle_graph(count, dir_name)
#print("Global time: {:.3f}".format(time.time() - ts_global_start - step_sleep * count))
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("\nDone.\n")
sys.exit(rc)
Output:
e:\Work\Dev\StackOverflow\q060876623> "e:\Work\Dev\VEnvs\py_pc064_03.07.06_test0\Scripts\python.exe" ./code00.py
Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)] 064bit on win32
OLD max open files: 512
NEW max open files: 513
draw_000.png
draw_001.png
draw_002.png
...
draw_167.png
draw_168.png
draw_169.png
Done.
Conclusions:
Apparently, some file handles (fds) are open, although they are not "seen" by TM or PE (probably they are on a lower level). However I don't know why this happens (is it a MS UCRT bug?), but from what I am concerned, once a child process ends, its streams should be closed, but I don't know how to force it (this would be a proper fix)
Also, the behavior (crash) when attempting to write (not open) to a fd (above the limit), seems a bit strange
As a workaround, the max open fds number can be increased. Based on the following inequality: 3 * (graph_count + 1) <= max_fds, you can get an idea about the numbers. From there, if you set the limit to 8192 (I didn't test this) you should be able handle 2729 graphs (assuming that there are no additional fds opened by the code)
Side notes:
While investigating, I ran into or noticed several adjacent issues, that I tried to fix:
Graphviz:
[GitLab]: graphviz/graphviz - [Issue #1481]: MSB4018 The NativeCodeAnalysis task failed unexpectedly. (merged on 200406)
PyGraphviz:
[GitHub]: pygraphviz/pygraphviz - AGraph Graphviz handle close mechanism (merged on 200720)
There's also an issue open for this behavior (probably the same author): [GitHub]: pygraphviz/pygraphviz - Pygraphviz crashes after drawing 170 graphs

I tried you code and it generated 200 graphs with no problem (I also tried with 2000).
My suggestion is to use these versions of the packages, I installed a conda environment on mac os with python 3.7 :
graphviz 2.40.1 hefbbd9a_2
pygraphviz 1.3 py37h1de35cc_1

Enumerating all modules for a binary using Python (pefile / win32api)

I want to use PEfile or another Python library to enumerate all modules. I thought I had it, but then I go into WinDbg because some obvious ones were missing, and I saw there were a number of missing ones.
For filezilla.exe:
00400000 00fe7000 image00400000 image00400000
01c70000 01ecc000 combase C:\WINDOWS\SysWOW64\combase.dll
6f590000 6f5ac000 SRVCLI C:\WINDOWS\SysWOW64\SRVCLI.DLL
6f640000 6f844000 COMCTL32 C:\WINDOWS\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17134.472_none_42ecd1cc44e43e73\COMCTL32.DLL
70610000 7061b000 NETUTILS C:\WINDOWS\SysWOW64\NETUTILS.DLL
70720000 70733000 NETAPI32 C:\WINDOWS\SysWOW64\NETAPI32.dll
72910000 72933000 winmmbase C:\WINDOWS\SysWOW64\winmmbase.dll
729d0000 729d8000 WSOCK32 C:\WINDOWS\SysWOW64\WSOCK32.DLL
72b40000 72b64000 WINMM C:\WINDOWS\SysWOW64\WINMM.DLL
72b70000 72b88000 MPR C:\WINDOWS\SysWOW64\MPR.DLL
73c60000 73c6a000 CRYPTBASE C:\WINDOWS\SysWOW64\CRYPTBASE.dll
73c70000 73c90000 SspiCli C:\WINDOWS\SysWOW64\SspiCli.dll
74120000 741b6000 OLEAUT32 C:\WINDOWS\SysWOW64\OLEAUT32.dll
741c0000 7477a000 windows_storage C:\WINDOWS\SysWOW64\windows.storage.dll
74780000 7487c000 ole32 C:\WINDOWS\SysWOW64\ole32.dll
74880000 74908000 shcore C:\WINDOWS\SysWOW64\shcore.dll
74910000 7498d000 msvcp_win C:\WINDOWS\SysWOW64\msvcp_win.dll
74990000 74a4f000 msvcrt C:\WINDOWS\SysWOW64\msvcrt.dll
74a50000 74a72000 GDI32 C:\WINDOWS\SysWOW64\GDI32.dll
74bd0000 74bde000 MSASN1 C:\WINDOWS\SysWOW64\MSASN1.dll
74be0000 74c47000 WS2_32 C:\WINDOWS\SysWOW64\WS2_32.dll
74c70000 74d30000 RPCRT4 C:\WINDOWS\SysWOW64\RPCRT4.dll
74d30000 74d37000 Normaliz C:\WINDOWS\SysWOW64\Normaliz.dll
74d40000 74d79000 cfgmgr32 C:\WINDOWS\SysWOW64\cfgmgr32.dll
74fe0000 75025000 powrprof C:\WINDOWS\SysWOW64\powrprof.dll
75150000 7526e000 ucrtbase C:\WINDOWS\SysWOW64\ucrtbase.dll
75280000 75416000 CRYPT32 C:\WINDOWS\SysWOW64\CRYPT32.dll
75420000 75584000 gdi32full C:\WINDOWS\SysWOW64\gdi32full.dll
755c0000 755c8000 FLTLIB C:\WINDOWS\SysWOW64\FLTLIB.DLL
755d0000 755e8000 profapi C:\WINDOWS\SysWOW64\profapi.dll
755f0000 75635000 SHLWAPI C:\WINDOWS\SysWOW64\SHLWAPI.dll
75640000 7698a000 SHELL32 C:\WINDOWS\SysWOW64\SHELL32.dll
76990000 76b74000 KERNELBASE C:\WINDOWS\SysWOW64\KERNELBASE.dll
76cf0000 76cff000 kernel_appcore C:\WINDOWS\SysWOW64\kernel.appcore.dll
76d00000 76d17000 win32u C:\WINDOWS\SysWOW64\win32u.dll
76db0000 76e86000 COMDLG32 C:\WINDOWS\SysWOW64\COMDLG32.DLL
76e90000 7701d000 USER32 C:\WINDOWS\SysWOW64\USER32.dll
77020000 77064000 sechost C:\WINDOWS\SysWOW64\sechost.dll
77100000 771e0000 KERNEL32 C:\WINDOWS\SysWOW64\KERNEL32.DLL
771e0000 77238000 bcryptPrimitives C:\WINDOWS\SysWOW64\bcryptPrimitives.dll
77240000 772b8000 ADVAPI32 C:\WINDOWS\SysWOW64\ADVAPI32.dll
773b0000 77540000 ntdll ntdll.dll
This is the output that I obtained from pefile by using a simlar script:
ADVAPI32.dll
COMCTL32.DLL
COMDLG32.DLL
CRYPT32.dll
GDI32.dll
KERNEL32.dll
MPR.DLL
msvcrt.dll
NETAPI32.dll
Normaliz.dll
ole32.dll
OLEAUT32.dll
POWRPROF.dll
SHELL32.DLL
USER32.dll
WINMM.DLL
WS2_32.dll
WSOCK32.DLL
def findDLL():
pe = pefile.PE(name)
for each in pe.DIRECTORY_ENTRY_IMPORT:
print entry.dll
Is there something else in Pefile I should be looking at to obtain more complete listing of modules that will be loaded?
Is there something in win32api or win32con that could get me this information? I would prefer pefile if possible, but either works. I need to be able to output a listing of all modules that would be loaded. I am working in Python and inflexible about changing.

Modules can be loaded into a process in many ways. Imported DLLs are just one way.
Imported DLLs may import DLLs for themselves too. foobar.exe may depend on user32.dll, but user32.dll in turn also depends on kernel32.dll so that will be loaded into your process. If you want a complete list, you may want to check the imported DLLs if your executable's dependencies.
Modules can be dynamically loaded in code with LoadLibrary(). You will not see those in the import directory. You'll have to disassemble the code for that and even then the library name can be generated on the fly so it will be hard to tell.
There are some more unsupported methods of loading modules that malware can use.
As the comments mentioned, getting a list of loaded modules through debugging APIs is probably simpler. But it all depends on what you're actually trying to do with this data.

As a matter of fact, there are different types of techniques to import modules. The type is determined by the way the referenced module is bound to the executable. As far as I know, PEfile does only lists the dynamic link libraries that are statically bound to the executable through the Imports table. Other types of dynamic link libraries are: explicit (those called via LoadLibrary/GetProcAddress APIs), Forwarded (those loaded via a PE mechanism that allow forwarding API calls) and delayed (those loaded via a PE mechanism that allow delayed loading of API calls).
Please find below a schema representing these methods (which is part of my slides available https://winitor.com/pdf/DynamicLinkLibraries.pdf
I hope that helps.

Portable way of detecting number of *usable* CPUs in Python

Per this question and answer -- Python multiprocessing.cpu_count() returns '1' on 4-core Nvidia Jetson TK1 -- the output of Python's multiprocessing.cpu_count() function on certain systems reflects the number of CPUs actively in use, as opposed to the number of CPUs actually usable by the calling Python program.
A common Python idiom is to use the return-value of cpu_count() to initialize the number of processes in a Pool. However, on systems that use such a "dynamic CPU activation" strategy, that idiom breaks rather badly (at least on a relatively quiescent system).
Is there some straightforward (and portable) way to get at the number of usable processors (as opposed the number currently in use) from Python?
Notes:
This question is not answered by the accepted answer to How to find out the number of CPUs using python, since as noted in the question linked at the top of this question, printing the contents of /proc/self/status shows all 4 cores as being available to the program.
To my mind, "portable" excludes any approach that involves parsing the contents of /proc/self/status, whose format may vary from release to release of Linux, and which doesn`t even exist on OS X. (The same goes for any other pseudo-file, as well.)

I don't think you will get any truly portable answers, so I will give a correct one.
The correct* answer for Linux is len(os.sched_getaffinity(pid)), where pid may be 0 for the current process. This function is exposed in Python 3.3 and later; if you need it in earlier, you'll have to do some fancy cffi coding.
Edit: you might try to see if you can use a function int omp_get_num_procs(); if it exists, it is the only meaningful answer I found on this question but I haven't tried it from Python.

Use psutil:
from the doc https://psutil.readthedocs.io/en/latest/:
>>> import psutil
>>> psutil.cpu_count()
4
>>> psutil.cpu_count(logical=False) # Ignoring virtual cores
2
This is portable

Here's an approach that gets the number of available CPU cores for the current process on systems that implement sched_getaffinity, and Windows:
import ctypes
import ctypes.wintypes
import os
from platform import system
def num_available_cores() -> int:
if hasattr(os, 'sched_getaffinity'):
return len(os.sched_getaffinity(0))
elif system() == 'Windows':
kernel32 = ctypes.WinDLL('kernel32')
DWORD_PTR = ctypes.wintypes.WPARAM
PDWORD_PTR = ctypes.POINTER(DWORD_PTR)
GetCurrentProcess = kernel32.GetCurrentProcess
GetCurrentProcess.restype = ctypes.wintypes.HANDLE
GetProcessAffinityMask = kernel32.GetProcessAffinityMask
GetProcessAffinityMask.argtypes = (ctypes.wintypes.HANDLE, PDWORD_PTR, PDWORD_PTR)
mask = DWORD_PTR()
if not GetProcessAffinityMask(GetCurrentProcess(), ctypes.byref(mask), ctypes.byref(DWORD_PTR())):
raise Exception("Call to 'GetProcessAffinityMask' failed")
return bin(mask.value).count('1')
else:
raise Exception('Cannot determine the number of available cores')
On Linux and any other systems that implement sched_getaffinity, we use Python's built-in wrapper for it.
On Windows we use ctypes to call GetProcessAffinityMask.
As far as I know there are no user APIs or tools to get/set the CPU affinity on macOS. In most cases os.cpu_count() will work fine, but if you truly need the number of available cores you may be out of luck.

Getting total/free RAM from within Python

From within a Python application, how can I get the total amount of RAM of the system and how much of it is currently free, in a cross-platform way?
Ideally, the amount of free RAM should consider only physical memory that can actually be allocated to the Python process.

Have you tried SIGAR - System Information Gatherer And Reporter?
After install
import os, sigar
sg = sigar.open()
mem = sg.mem()
sg.close()
print mem.total() / 1024, mem.free() / 1024
Hope this helps

psutil would be another good choice. It also needs a library installed however.
>>> import psutil
>>> psutil.virtual_memory()
vmem(total=8374149120L, available=2081050624L, percent=75.1,
used=8074080256L, free=300068864L, active=3294920704,
inactive=1361616896, buffers=529895424L, cached=1251086336)

For the free memory part, there is a function in the wx library:
wx.GetFreeMemory()
Unfortunately, this only works on Windows. Linux and Mac ports either return "-1" or raise a NotImplementedError.

You can't do this with just the standard Python library, although there might be some third party package that does it. Barring that, you can use the os package to determine which operating system you're on and use that information to acquire the info you want for that system (and encapsulate that into a single cross-platform function).

In windows I use this method. It's kinda hacky but it works using standard os library:
import os
process = os.popen('wmic memorychip get capacity')
result = process.read()
process.close()
totalMem = 0
for m in result.split(" \r\n")[1:-1]:
totalMem += int(m)
print totalMem / (1024**3)