Develop Reference

Pygraphviz crashes after drawing 170 graphs

I am using pygraphviz to create a large number of graphs for different configurations of data. I have found that no matter what information is put in the graph the program will crash after drawing the 170th graph. There are no error messages generated the program just stops. Is there something that needs to be reset if drawing this many graphs?
I am running Python 3.7 on a Windows 10 machine, Pygraphviz 1.5, and graphviz 2.38
for graph_number in range(200):
config_graph = pygraphviz.AGraph(strict=False, directed=False, compound=True, ranksep='0.2', nodesep='0.2')
# Create Directory
if not os.path.exists('Graph'):
os.makedirs('Graph')
# Draw Graph
print('draw_' + str(graph_number))
config_graph.layout(prog = 'dot')
config_graph.draw('Graph/'+str(graph_number)+'.png')

I was able to constantly reproduce the behavior with:
Python 3.7.6 (pc064 (64bit), then also with pc032)
PyGraphviz 1.5 (that I built - available for download at [GitHub]: CristiFati/Prebuilt-Binaries - Various software built on various platforms. (under PyGraphviz, naturally).
Might also want to check [SO]: Installing pygraphviz on Windows 10 64-bit, Python 3.6 (#CristiFati's answer))
Graphviz 2.42.2 ((pc032) same as #2.)
I suspected an Undefined Behavior somewhere in the code, even if the behavior was precisely the same:
OK for 169 graphs
Crash for 170
Did some debugging (added some print(f) statements in agraph.py, and cgraph.dll (write.c)).
PyGraphviz invokes Graphviz's tools (.exes) for many operations. For that, it uses subprocess.Popen and communicates with the child process via its 3 available streams (stdin, stdout, stderr).
From the beginning I noticed that 170 * 3 = 510 (awfully close to 512 (0x200)), but didn't pay as much attention as I should have until later (mostly because the Python process (running the code below) had no more than ~150 open handles in Task Manager (TM) and also Process Explorer (PE)).
However, a bit of Googleing revealed:
[SO]: Is there a limit on number of open files in Windows (#stackprogrammer's answer) (and from here)
[MS.Learn]: _setmaxstdio (which states (emphasis is mine)):
C run-time I/O now supports up to 8,192 files open simultaneously at the low I/O level. This level includes files opened and accessed using the _open, _read, and _write family of I/O functions. By default, up to 512 files can be open simultaneously at the stream I/O level. This level includes files opened and accessed using the fopen, fgetc, and fputc family of functions. The limit of 512 open files at the stream I/O level can be increased to a maximum of 8,192 by use of the _setmaxstdio function.
[SO]: Python: Which command increases the number of open files on Windows? (#NorthCat's answer)
Below is your code that I modified for debugging and reproducing the error. It needs (for code shortness' sake, as same thing can be achieved via CTypes) the PyWin32 package (python -m pip install pywin32).
code00.py:
#!/usr/bin/env python
import os
import sys
#import time
import pygraphviz as pgv
import win32file as wfile
def handle_graph(idx, dir_name):
graph_name = "draw_{:03d}".format(idx)
graph_args = {
"name": graph_name,
"strict": False,
"directed": False,
"compound": True,
"ranksep": "0.2",
"nodesep": "0.2",
}
graph = pgv.AGraph(**graph_args)
# Draw Graph
img_base_name = graph_name + ".png"
print(" {:s}".format(img_base_name))
graph.layout(prog="dot")
img_full_name = os.path.join(dir_name, img_base_name)
graph.draw(img_full_name)
graph.close() # !!! Has NO (visible) effect, but I think it should be called anyway !!!
def main(*argv):
print("OLD max open files: {:d}".format(wfile._getmaxstdio()))
# 513 is enough for your original code (170 graphs), but you can set it up to 8192
#wfile._setmaxstdio(513) # !!! COMMENT this line to reproduce the crash !!!
print("NEW max open files: {:d}".format(wfile._getmaxstdio()))
dir_name = "Graph"
# Create Directory
if not os.path.isdir(dir_name):
os.makedirs(dir_name)
#ts_global_start = time.time()
start = 0
count = 170
#count = 1
step_sleep = 0.05
for i in range(start, start + count):
#ts_local_start = time.time()
handle_graph(i, dir_name)
#print(" Time: {:.3f}".format(time.time() - ts_local_start))
#time.sleep(step_sleep)
handle_graph(count, dir_name)
#print("Global time: {:.3f}".format(time.time() - ts_global_start - step_sleep * count))
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("\nDone.\n")
sys.exit(rc)
Output:
e:\Work\Dev\StackOverflow\q060876623> "e:\Work\Dev\VEnvs\py_pc064_03.07.06_test0\Scripts\python.exe" ./code00.py
Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)] 064bit on win32
OLD max open files: 512
NEW max open files: 513
draw_000.png
draw_001.png
draw_002.png
...
draw_167.png
draw_168.png
draw_169.png
Done.
Conclusions:
Apparently, some file handles (fds) are open, although they are not "seen" by TM or PE (probably they are on a lower level). However I don't know why this happens (is it a MS UCRT bug?), but from what I am concerned, once a child process ends, its streams should be closed, but I don't know how to force it (this would be a proper fix)
Also, the behavior (crash) when attempting to write (not open) to a fd (above the limit), seems a bit strange
As a workaround, the max open fds number can be increased. Based on the following inequality: 3 * (graph_count + 1) <= max_fds, you can get an idea about the numbers. From there, if you set the limit to 8192 (I didn't test this) you should be able handle 2729 graphs (assuming that there are no additional fds opened by the code)
Side notes:
While investigating, I ran into or noticed several adjacent issues, that I tried to fix:
Graphviz:
[GitLab]: graphviz/graphviz - [Issue #1481]: MSB4018 The NativeCodeAnalysis task failed unexpectedly. (merged on 200406)
PyGraphviz:
[GitHub]: pygraphviz/pygraphviz - AGraph Graphviz handle close mechanism (merged on 200720)
There's also an issue open for this behavior (probably the same author): [GitHub]: pygraphviz/pygraphviz - Pygraphviz crashes after drawing 170 graphs

I tried you code and it generated 200 graphs with no problem (I also tried with 2000).
My suggestion is to use these versions of the packages, I installed a conda environment on mac os with python 3.7 :
graphviz 2.40.1 hefbbd9a_2
pygraphviz 1.3 py37h1de35cc_1

Stop new gdal command window opening each time gdal command is executed python

I have written a PyQGIS script that uses gdals warp. The piece of code doing this for me is as follows:
warp = 'gdalwarp -ot Byte -q -of GTiff -tr 2.81932541777e-05 -2.81932541777e-05 -tap -cutline %s -crop_to_cutline -co COMPRESS=DEFLATE -co PREDICTOR=1 -co ZLEVEL=6 -wo OPTIMIZE_SIZE=TRUE %s %s' % (instrv, ('"' + pathsplitedit + '"'), outputpath2)
call (warp)
So I have this in a loop and all is good. However each time it executes a new command window is opened, which isn't ideal as it loops through up 100 features in a shapefile. Is there a way I can not have the command window open at all? Any help is really appreciated!!

Since GDAL 2.1 (which recent versions of QGIS use) you can access the command line utilities from the bindings itself, which has a lot of benefits.
Your call becomes something like this, notice that i didn't copy all your creation options, its just to give you an idea on how to use it.
warpopts = gdal.WarpOptions(outputType=gdal.GDT_Byte,
format='GTiff',
xRes=2.81932541777e-05,
yRes=-2.81932541777e-05,
cutlineDSName='cutline_vec.shp',
cropToCutline=True,
targetAlignedPixels=True,
options=['COMPRESS=DEFLATE'])
ds = gdal.Warp('raster_out.tif', 'raster_in.tif', options=warpopts)
ds = None
One of the benefits is that the input files don't have to be on-disk but can also be opened gdal/ogr Datasets. The gdal.Warp cmd also returns the output file as an opened Dataset, which you can then pass on to other commands.

Python run .exe with input

I would like to use a script to call an executable program and input some instructions in the exe program (a DOS window) to run this exe program (output not required)
For example if I directly run the program I'll double click it and type as follows:
load XXX.txt
oper
quit
Here's my code, fwiw I do not have a deep understanding about subprocess.
import subprocess
import os
os.chdir('D:/Design/avl3.35/Avl/Runs')
Process=subprocess.Popen(['avl.exe'], stdin=subprocess.PIPE)
Process.communicate(b'load allegro.avl\n')
When I run this code, I get the following:
===================================================
Athena Vortex Lattice Program Version 3.35
Copyright (C) 2002 Mark Drela, Harold Youngren
This software comes with ABSOLUTELY NO WARRANTY,
subject to the GNU General Public License.
Caveat computor
===================================================
==========================================================
Quit Exit program
.OPER Compute operating-point run cases
.MODE Eigenvalue analysis of run cases
.TIME Time-domain calculations
LOAD f Read configuration input file
MASS f Read mass distribution file
CASE f Read run case file
CINI Clear and initialize run cases
MSET i Apply mass file data to stored run case(s)
.PLOP Plotting options
NAME s Specify new configuration name
AVL c>
Reading file: allegro.avl ...
Configuration: Allegro-lite 2M
Building surface: WING
Reading airfoil from file: ag35.dat
Reading airfoil from file: ag36.dat
Reading airfoil from file: ag37.dat
Reading airfoil from file: ag38.dat
At line 145 of file ../src/userio.f (unit = 5, file = 'stdin') #!!!!Error here!!
Fortran runtime error: End of file #!!!!Error here!!
Building duplicate image-surface: WING (YDUP)
Building surface: Horizontal tail
Building duplicate image-surface: Horizontal tail (YDUP)
Building surface: Vertical tail
Mach = 0.0000 (default)
Nbody = 0 Nsurf = 5 Nstrp = 64 Nvor = 410
Initializing run cases...
I have no idea what's wrong with this, nor do I know why the error prompt shows inside the codes. After searching I see that communicate method is to wait for the process to finish and return all the output, though I do not need output, but I still don't know what to do.
Could you explain what is happening here and how could I finish what I want to do?

matplotlib.animation: script works under Windows, and works under Linux for short animations, but hangs indefinitely for longer animations?

Here is the relevant code snippet:
ani = animation.FuncAnimation(fig, animate, outputFiles, blit=True, interval=25)
print "Starting to write animation file..."
# Set up formatting for the movie files
Writer = animation.writers['ffmpeg']
fps = 2
animationFileName = "animation"
if fps <= 2:
animationFileName = animationFileName + "_slow.mp4"
if fps < 120:
animationFileName = animationFileName + ".mp4"
else:
animationFileName = animationFileName + "_fast.mp4"
writer = Writer(fps=fps, metadata=dict(artist='Alien'), bitrate=1800)
ani.save(os.path.join(graphicsOutputDirectory, animationFileName), writer=writer)
print "Finished writing animation file..."
When running the script in Linux (Debian Wheezy), I see the following output:
>>> python make_animation.py
Starting to run animation file...
In Windows, the script runs just fine. I have made sure that I am not making silly errors like using incorrect file names. Short animations seem to run just fine on my GNU/Linux machine and longer more serious ones seem to hang indefinitely...
What might the issue here be? I suppose a relevant detail I should mention is that I am using a virtualenvironment to run an up-to-date version of matplotlib.

Ubuntu and Debian now ship with deprecated FFMPEG versions. That may be the issue. Try replacing the ffmpeg binary in your /usr/bin by a recent one from the FFMPEG website.
Another possible issue concerns Python 3, in which subprocess has a smaller buffer and will hang forever if ffmpeg sends too much informations back to Python. Are you on Python 3 ?

Python: what are the nearest Linux and OS X [edit: macOS] equivalents of winsound.Beep?

If one wishes to beep the speaker on Windows, Python 2 apparently provides a useful function: winsound.Beep(). The neat thing about this function is that it takes arguments specifying the exact frequency and duration of the beep. This is exactly what I want to do, except that I don't use Windows. So...
What are the nearest equivalents of winsound.Beep() for Linux and OS X [edit: macOS], bringing in as few dependencies as possible?
Please note that I want to be able to beep the speaker directly, not to play a sound file. Also, I need to be able to control the frequency and duration of the beep, so curses.beep() and print '\a' won't do. Lastly, I am aware that PyGame provides extensive sound capabilities, but given that I don't require any of PyGame's other functionality, that would seem like using a sledgehammer to crack a nut (and anyway, I'm trying to do away with dependencies as far as possible).
[Edited on 9 Feb 2023 to reflect the fact that OS X was renamed macOS a few years after this question was asked]

winsound is only for windows and I could not find any cross platform way to do this, other than print "/a". However, you cannot set the frequency and duration with this.
However, you can try the os.system command to do the same with the system command beep. Here is a snippet, which defines the function playsound in a platform independent way
try:
import winsound
except ImportError:
import os
def playsound(frequency,duration):
#apt-get install beep
os.system('beep -f %s -l %s' % (frequency,duration))
else:
def playsound(frequency,duration):
winsound.Beep(frequency,duration)
For more info, look at this blog
EDIT: You will need to install the beep package on linux to run the beep command. You can install by giving the command
sudo apt-get install beep

I found a potential solution here:
http://bytes.com/topic/python/answers/25217-beeping-under-linux
It involves writing directly to /dev/audio. Not sure how portable it is or if it even works at all - i'm not on a linux machine atm.
def beep(frequency, amplitude, duration):
sample = 8000
half_period = int(sample/frequency/2)
beep = chr(amplitude)*half_period+chr(0)*half_period
beep *= int(duration*frequency)
audio = file('/dev/audio', 'wb')
audio.write(beep)
audio.close()

This works on mac:
import numpy as np
import simpleaudio as sa
def sound(x,z):
frequency = x # Our played note will be 440 Hz
fs = 44100 # 44100 samples per second
seconds = z # Note duration of 3 seconds
# Generate array with seconds*sample_rate steps, ranging between 0 and seconds
t = np.linspace(0, seconds, seconds * fs, False)
# Generate a 440 Hz sine wave
note = np.sin(frequency * t * 2 * np.pi)
# Ensure that highest value is in 16-bit range
audio = note * (2**15 - 1) / np.max(np.abs(note))
# Convert to 16-bit data
audio = audio.astype(np.int16)
# Start playback
play_obj = sa.play_buffer(audio, 1, 2, fs)
# Wait for playback to finish before exiting
play_obj.wait_done()
sound(300,2)
sound(200,1)

The most light-weight cross-platform layer I can see is "PortAudio". This is used by R for instance in their package to wrap platform-specific driver calls into simple play/record of digitized waveforms as an array.
The good folk at M.I.T. produce a Python binding for this, but you will need to include the compiled .dll/.so for this to work. http://people.csail.mit.edu/hubert/pyaudio/
( libao is similar by Xiph the makers of Ogg/Vorbis , a wrapper pyao exists but this seems less widely used )
SoX is an excellent set of cross-platform tools with much more functionality for format conversion and reading files etc..
Using ctypes to make calls from Python to a driver is feasible but very messy, even the simplest legacy WinMM.

I've found 3 methods for Linux:
new method using the Linux evdev API, works with any user in the input group (example source code)
old method using fcntl and /dev/console (requires root priviledges) (example source code)
invoke the beep command directly with subprocess or os.system (slower and must be installed in the system).
See also my tone() function here with all the alternatives.