I am working with WARC Tools, a python library for working with WARC files. I've installed everything and it all seems to work except for the last command. While I suspect many of you may not have worked with this tool, perhaps you can make sense of the error message for me.
When I run this command:
python /path/filesdump.py filtered.warc as per the documentation, I get this error message immediately:
html/811cac8c-7430-403b-96a4-7d77137b0d46.html
Traceback (most recent call last):
File "/users/ianmilligan1/desktop/warc/warc-tools-mandel/filesdump.py", line 63, in <module>
sys.exit(main(sys.argv))
File "/users/ianmilligan1/desktop/warc/warc-tools-mandel/filesdump.py", line 34, in main
dump_archive(fh,name)
File "/users/ianmilligan1/desktop/warc/warc-tools-mandel/filesdump.py", line 57, in dump_archive
txt = record.filedump(content=True)
File "/Users/ianmilligan1/Desktop/WARC/warc-tools-mandel/warctools/record.py", line 140, in filedump
p = Popen(['lynx', '-dump', '-stdin', '-nomargins', '-unique_urls', '-width=120'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
What's going on here? Where would you suggest I begin looking?
You need to install lynx, a command-line web browser that WARC Tools seems to require. That'll solve it.
What's going on is that warctools/record.py is calling lynx on line 140, and Python's subprocess library isn't finding the lynx executable so it throws an exception. They really ought to wrap the exception and provide a clearer message... Anyways. Try that and see if it fixes it.
Related
I have a Raspberry PI and I want to automatically open a website at boot time.
So I made an auto start config to run a python (v.3.5.3) script on boot.
The python script looks like this:
import os
import subprocess
subprocess.call([/home/pi/Desktop/Shortcut Final.desktop])
But I get the error:
Traceback (most recent call last): File "/home/pi/myscript.py", line 3
subprocess.call([/home/pi/Desktop/Shortcut Final.desktop])
^ SyntaxError: invalid syntax
I got my instructions from a guide that used windows file location, so I think the error is that my file location dont start with a directory letter like C:, but I dont think raspbian use the same system.
I have not found any other guides that explain how to accomplish the same thing in simple terms (as i only know very basic python coding)
I have looked into the open() command, but it seemed complicated.
Any tips on how to accomplish my goal/solve the error?
After putting the file location in quotes the syntax error disappears and i encounter a new error:
Traceback (most recent call last):
File "/home/pi/myscript.py", line 5, in <module>
subprocess.call(["/home/pi/Desktop/Shortcut Final.desktop"])
File "/usr/lib/python3.5/subprocess.py", line 247, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1282, in _execute_child
raise child_exception_type(errno_num, err_msg)
PermissionError: [Errno 13] Permission denied
I have been trying to install the mlabwrap Python library, but keep on running into the following error when I run the setup.py file using the command python setup.py install in the terminal:
Traceback (most recent call last):
File "setup.py", line 130, in <module>
queried_version, queried_dir, queried_platform_dir = matlab_params(cmd, WINDOWS, extra_args)
File "setup.py", line 93, in matlab_params
error = call(cmd, **extra_args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I have done some investigation, and found the following forum post: https://sourceforge.net/p/mlabwrap/mailman/message/29077359/ to resolve the issue. Following the advice in this thread, I changed the following things in the setup.py file:
__version__ = '7.3'
and
MATLAB_DIR = '/Applications/MATLAB_R2015a.app'
I have a feeling that the problem is related to finding the directory that matlab is located in, but I seem to be unable to resolve this. I didn't change my $PATH variable because I was a little confused by the advice given in the thread. Any insight would be much appreciated!
In order to install mlabwrap, you must modify the setup.py file to:
MATLAB_COMMAND = '/Applications/MATLAB_R2015a.app/bin/matlab' # specify a full path if not in PATH
MATLAB_VERSION = 7.3 # e.g: 6 (one of (6, 6.5, 7, 7.3))
# 7.3 includes later versions as well
MATLAB_DIR= '/Applications/MATLAB_R2015a.app/' # e.g: '/usr/local/matlab'; 'c:/matlab6'
PLATFORM_DIR='maci64' # e.g: 'glnx86'; r'win32/microsoft/msvc60'
EXTRA_COMPILE_ARGS=None # e.g: ['-G']
The parameters may change depending on what version of MATLAB you are using and what operating system you are using.
I'm working with a set of scripts aimed at modifying an android app, the github repo located here: Ingress Broot Mod. I installed Python 3.2, then installed PyYAML 3.10. I moved on to follow the build instructions and "Import Ingress apk using import_apk.py script."
Specifically, I typed: B:\GitHub\ingress-apk-mod\bin>python import_apk.py "B:\com.nianticproject.ingress-1.apk"
At which point this error pops up.
Traceback (most recent call last):
File "B:\GitHub\ingress-apk-mod\bin\import_apk.py", line 28, in <module>
main(sys.argv[1])
File "B:\GitHub\ingress-apk-mod\bin\import_apk.py", line 24, in main
''' % (LINE_PREFIX, apk), shell=True, executable='/bin/bash')
File "C:\Python32\lib\subprocess.py", line 483, in check_call
retcode = call(*popenargs, **kwargs)
File "C:\Python32\lib\subprocess.py", line 470, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Python32\lib\subprocess.py", line 744, in __init__
restore_signals, start_new_session)
File "C:\Python32\lib\subprocess.py", line 977, in _execute_child
startupinfo)
WindowsError: [Error 3] The system cannot find the path specified
The closest I could find was Unable to run call subprocess method using python 3.2.
Is this an easy fix; if not, what steps should I take to fix this problem?
This is my first time using python - could this possibly stem from improperly installed Python (did I miss a step?)
Questions with similar issue:
Parallel Python - too many files and Python too many open files (subprocesses)
I am using Parallel Python [V1.6.2] to run tasks. The task processes an input file and outputs a log/report. Say, there are 10 folders each with 5000 ~ 20000 files which are read in parallel, processed and logs written out. Each file is approx 50KB ~ 250KB
After ~6 Hours of running, Parallel Python fails with the following error.
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 342, in __init__
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 506, in set_ncpus
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 140, in __init__
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 146, in start
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
File "/usr/lib/python2.7/subprocess.py", line 1135, in _execute_child
File "/usr/lib/python2.7/subprocess.py", line 1091, in pipe_cloexec
OSError: [Errno 24] Too many open files
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook
ImportError: No module named fileutils
Original exception was:
Traceback (most recent call last):
File "PARALLEL_TEST.py", line 746, in <module>
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 342, in __init__
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 506, in set_ncpus
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 140, in __init__
File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 146, in start
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
File "/usr/lib/python2.7/subprocess.py", line 1135, in _execute_child
File "/usr/lib/python2.7/subprocess.py", line 1091, in pipe_cloexec
OSError: [Errno 24] Too many open files
While I understand, this could be the issue in subprocess pointed out here http://bugs.python.org/issue2320, but, seems the solution is only part of Py V3.2. I am currently tied to Py V2.7.
I would like to know if the following suggestion helps:
[1]http://www.parallelpython.com/component/option,com_smf/Itemid,1/topic,313.0
*) Adding worker.t.close() in destroy() method of /usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py
*) Increasing BROADCAST_INTERVAL in /usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/ppauto.py
I would like to know if there is a fix available/Work Around for this issue in Python V2.7.
Thanks in Advance
My team recently stumbled upon a similar issue with the same file handle resource exhaustion issue while running celeryd task queue jobs. I believe the OP has nailed it and it's most likely the messy code in suprocess.py lib in Python 2.7 and Python 3.1.
As suggested in , Python Bug#2320, please pass in close_fds=True everywhere you call subprocess.Popen(). In fact they make that a default in Python 3.2 while also fixing the underlying race condition issue. See more details in that ticket.
I had left at some lines to destroy the job servers. job_server.destroy() fixes the issue.
I am trying to build adblockplus from this link. When I issue the command:
python build.py build
I end up getting the following error message. I tried with and without sudo to no avail.
Traceback (most recent call last):
File "build.py", line 10, in <module>
buildtools.build.processArgs('.', sys.argv)
File "/home/machine/projects/weird/buildtools/build.py", line 352, in processArgs
commands[command](baseDir, scriptName, opts, args, type)
File "/home/machine/projects/weird/buildtools/build.py", line 39, in __call__
return self._handler(baseDir, scriptName, opts, args, type)
File "/home/machine/projects/weird/buildtools/build.py", line 166, in runBuild
limitMetadata=limitMetadata)
File "/home/machine/projects/weird/buildtools/packager.py", line 274, in createBuild
buildNum = getBuildNum(baseDir)
File "/home/machine/projects/weird/buildtools/packager.py", line 80, in getBuildNum
(result, dummy) = subprocess.Popen(['hg', 'id', '-n'], stdout=subprocess.PIPE).communicate()
File "/usr/lib/python2.6/subprocess.py", line 623, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1141, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Am I missing a package? Any hint on what the problem could be will be much appreciated.
The stack trace makes it clear that the issue is caused by the following line:
(result, dummy) = subprocess.Popen(['hg', 'id', '-n'], stdout=subprocess.PIPE).communicate()
If you look at the subprocess package documentation, this line is trying to run the hg id -n command (get numerical Mercurial revision). Apparently, the Mercurial command line tool isn't present on your system (a possibility that this build script didn't consider) so it fails.
Disclaimer: I happen to be the one who wrote this script and I fixed this bug now. Mercurial isn't essential for the build, the revision number is pretty much only necessary to determine the output file name.