Questions with similar issue:
Parallel Python - too many files and Python too many open files (subprocesses)
I am using Parallel Python [V1.6.2] to run tasks. The task processes an input file and outputs a log/report. Say, there are 10 folders each with 5000 ~ 20000 files which are read in parallel, processed and logs written out. Each file is approx 50KB ~ 250KB
After ~6 Hours of running, Parallel Python fails with the following error.
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 342, in __init__
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 506, in set_ncpus
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 140, in __init__
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 146, in start
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
File "/usr/lib/python2.7/subprocess.py", line 1135, in _execute_child
File "/usr/lib/python2.7/subprocess.py", line 1091, in pipe_cloexec
OSError: [Errno 24] Too many open files
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook
ImportError: No module named fileutils
Original exception was:
Traceback (most recent call last):
File "PARALLEL_TEST.py", line 746, in <module>
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 342, in __init__
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 506, in set_ncpus
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 140, in __init__
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 146, in start
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
File "/usr/lib/python2.7/subprocess.py", line 1135, in _execute_child
File "/usr/lib/python2.7/subprocess.py", line 1091, in pipe_cloexec
OSError: [Errno 24] Too many open files
While I understand, this could be the issue in subprocess pointed out here http://bugs.python.org/issue2320, but, seems the solution is only part of Py V3.2. I am currently tied to Py V2.7.
I would like to know if the following suggestion helps:
[1]http://www.parallelpython.com/component/option,com_smf/Itemid,1/topic,313.0
*) Adding worker.t.close() in destroy() method of /usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py
*) Increasing BROADCAST_INTERVAL in /usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/ppauto.py
I would like to know if there is a fix available/Work Around for this issue in Python V2.7.
Thanks in Advance
My team recently stumbled upon a similar issue with the same file handle resource exhaustion issue while running celeryd task queue jobs. I believe the OP has nailed it and it's most likely the messy code in suprocess.py lib in Python 2.7 and Python 3.1.
As suggested in , Python Bug#2320, please pass in close_fds=True everywhere you call subprocess.Popen(). In fact they make that a default in Python 3.2 while also fixing the underlying race condition issue. See more details in that ticket.
I had left at some lines to destroy the job servers. job_server.destroy() fixes the issue.
Related
I am having an issue with a program, which uses panda3d. It works perfectly when executed as a pythonscript, but the version, which is compiled (or rather packaged) by PyInstaller throws the exception below.
:display(warning): Unable to load libpandagl.so: No error.
Known pipe types:
(all display modules loaded.)
Traceback (most recent call last):
File "Shipsim3d_1-1.py", line 930, in <module>
File "Shipsim3d_1-1.py", line 23, in __init__
File "direct/showbase/ShowBase.py", line 339, in __init__
File "direct/showbase/ShowBase.py", line 1024, in openDefaultWindow
File "direct/showbase/ShowBase.py", line 1059, in openMainWindow
File "direct/showbase/ShowBase.py", line 769, in openWindow
File "direct/showbase/ShowBase.py", line 749, in <lambda>
File "direct/showbase/ShowBase.py", line 821, in _doOpenWindow
File "direct/showbase/ShowBase.py", line 650, in makeDefaultPipe
File "direct/directnotify/Notifier.py", line 130, in error
Exception: No graphics pipe is available!
Your Config.prc file must name at least one valid panda display
library via load-display or aux-display.
[5466] Failed to execute script 'Shipsim3d_1-1' due to unhandled exception!
The .prc file doesn't exist in the executable, but even if I copy it there manually, it still doesn't work. In my Code, I use a function called loadPrcData. Is that a Part of the problem? Or is it something else?
Thanks in advance
I am having an issue with a program, which uses panda3d. It works perfectly when executed as a pythonscript, but the version, which is compiled (or rather packaged) by PyInstaller throws the exception below.
:display(warning): Unable to load libpandagl.so: No error.
Known pipe types:
(all display modules loaded.)
Traceback (most recent call last):
File "Shipsim3d_1-1.py", line 930, in <module>
File "Shipsim3d_1-1.py", line 23, in __init__
File "direct/showbase/ShowBase.py", line 339, in __init__
File "direct/showbase/ShowBase.py", line 1024, in openDefaultWindow
File "direct/showbase/ShowBase.py", line 1059, in openMainWindow
File "direct/showbase/ShowBase.py", line 769, in openWindow
File "direct/showbase/ShowBase.py", line 749, in <lambda>
File "direct/showbase/ShowBase.py", line 821, in _doOpenWindow
File "direct/showbase/ShowBase.py", line 650, in makeDefaultPipe
File "direct/directnotify/Notifier.py", line 130, in error
Exception: No graphics pipe is available!
Your Config.prc file must name at least one valid panda display
library via load-display or aux-display.
[5466] Failed to execute script 'Shipsim3d_1-1' due to unhandled exception!
Usually, there is a Config.prc file in the etc subdirectory of the panda3d site-package directory. This file is non-existent in the packaged version. But if I create this subdirectory there, and copy the files to it as well, it still doesn't work and it still throws the same exception. There is a line "loadPrcData("win-size 1080 1920")" in my main program as well. Could this be part of the problem in any way? Or is there something else i am missing?
Thanks in advance
I solved this problem by including the whole panda3d lib:
pyinstaller --add-data="path/to/panda3d;panda3d" -wF main.py
I'm using ubuntu 13.10 running in a VM on OSX, python2.7 and GAE 1.8.8.
Lauching dev_appserver.py results in the following error:
INFO 2013-12-10 03:53:30,046 api_server.py:527] Saving search indexes
Traceback (most recent call last):
File "/home/ubuntu/xxxxxx/google_appengine/dev_appserver.py", line 197, in <module>
File "/home/ubuntu/xxxxxx/google_appengine/dev_appserver.py", line 193, in _run_file
File "/home/ubuntu/xxxxxx/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 872, in <module>
File "/home/ubuntu/xxxxxx/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 868, in main
File "/home/ubuntu/xxxxxx/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 707, in stop
File "/home/ubuntu/xxxxxx/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 141, in quit
File "/home/ubuntu/xxxxxx/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 528, in cleanup_stubs
File "/home/ubuntu/xxxxxx/google_appengine/google/appengine/api/search/simple_search_stub.py", line 984, in Write
File "/usr/lib/python2.7/tempfile.py", line 304, in mkstemp
File "/usr/lib/python2.7/tempfile.py", line 239, in _mkstemp_inner
OSError: [Errno 24] Too many open files: '/tmp/appengine.xxxxxx-hr-dev.ubuntu/tmpMVVXrH'
Any ideas?
Check the shared memory parameter, kern.sysv.shmseg on your linux system and set it right by increasing it.
To view the shared memory parameters, use:
sysctl -A | grep shm
To update that parameters, edit file:
sudo nano /etc/sysctl.conf
Refer to this SO answer for more information.
I am working with WARC Tools, a python library for working with WARC files. I've installed everything and it all seems to work except for the last command. While I suspect many of you may not have worked with this tool, perhaps you can make sense of the error message for me.
When I run this command:
python /path/filesdump.py filtered.warc as per the documentation, I get this error message immediately:
html/811cac8c-7430-403b-96a4-7d77137b0d46.html
Traceback (most recent call last):
File "/users/ianmilligan1/desktop/warc/warc-tools-mandel/filesdump.py", line 63, in <module>
sys.exit(main(sys.argv))
File "/users/ianmilligan1/desktop/warc/warc-tools-mandel/filesdump.py", line 34, in main
dump_archive(fh,name)
File "/users/ianmilligan1/desktop/warc/warc-tools-mandel/filesdump.py", line 57, in dump_archive
txt = record.filedump(content=True)
File "/Users/ianmilligan1/Desktop/WARC/warc-tools-mandel/warctools/record.py", line 140, in filedump
p = Popen(['lynx', '-dump', '-stdin', '-nomargins', '-unique_urls', '-width=120'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
What's going on here? Where would you suggest I begin looking?
You need to install lynx, a command-line web browser that WARC Tools seems to require. That'll solve it.
What's going on is that warctools/record.py is calling lynx on line 140, and Python's subprocess library isn't finding the lynx executable so it throws an exception. They really ought to wrap the exception and provide a clearer message... Anyways. Try that and see if it fixes it.
I am trying to build adblockplus from this link. When I issue the command:
python build.py build
I end up getting the following error message. I tried with and without sudo to no avail.
Traceback (most recent call last):
File "build.py", line 10, in <module>
buildtools.build.processArgs('.', sys.argv)
File "/home/machine/projects/weird/buildtools/build.py", line 352, in processArgs
commands[command](baseDir, scriptName, opts, args, type)
File "/home/machine/projects/weird/buildtools/build.py", line 39, in __call__
return self._handler(baseDir, scriptName, opts, args, type)
File "/home/machine/projects/weird/buildtools/build.py", line 166, in runBuild
limitMetadata=limitMetadata)
File "/home/machine/projects/weird/buildtools/packager.py", line 274, in createBuild
buildNum = getBuildNum(baseDir)
File "/home/machine/projects/weird/buildtools/packager.py", line 80, in getBuildNum
(result, dummy) = subprocess.Popen(['hg', 'id', '-n'], stdout=subprocess.PIPE).communicate()
File "/usr/lib/python2.6/subprocess.py", line 623, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1141, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Am I missing a package? Any hint on what the problem could be will be much appreciated.
The stack trace makes it clear that the issue is caused by the following line:
(result, dummy) = subprocess.Popen(['hg', 'id', '-n'], stdout=subprocess.PIPE).communicate()
If you look at the subprocess package documentation, this line is trying to run the hg id -n command (get numerical Mercurial revision). Apparently, the Mercurial command line tool isn't present on your system (a possibility that this build script didn't consider) so it fails.
Disclaimer: I happen to be the one who wrote this script and I fixed this bug now. Mercurial isn't essential for the build, the revision number is pretty much only necessary to determine the output file name.