Related
I am completely new to the subprocess module. And I was trying to automate the deauthentication attack commands. When I run airodump-ng wlan0mon as you know it looks for the APs nearby and the connected clients to it.
Now when I try to run this command using lets suppose p = subprocess.run(["airmon-ng","wlan0mon"], capture_output=True) in Python as you know this command runs until the user hits Ctrl+C, so it should save the last output when user hits Ctrl+C in the variable but instead I get error which is this:
Traceback (most recent call last):
File "Deauth.py", line 9, in <module>
p3 = subprocess.run(["airodump-ng","wlan0"], capture_output=True)
File "/usr/lib/python3.8/subprocess.py", line 491, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.8/subprocess.py", line 1024, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.8/subprocess.py", line 1866, in _communicate
ready = selector.select(timeout)
File "/usr/lib/python3.8/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt
What can I try to resolve this?
Just use Python's error handling. Catch any KeyboardInnterrupts (within your subprocess function) using try and except statements like so:
def stuff(things):
try:
# do stuff
except KeyboardInterrupt:
return last_value
I am trying to use Python to interact with another program via the command line. The main problem I am having is a specific call that has multiple follow-up prompts. Initially the command line call asks for the name of a project and then proceeds to ask if I would like to view any of the subfolders of the project. I need to answer y/n to each of these in order and the answer to each is unfortunately not all y or n. Additionally, I cannot know the answer to the question without reading the individual prompts so I am incapable of sending a block of 'y's or 'n's all at once.
This is the command line call:
si viewproject
After entering the command, the command line prompts:
Enter the project name:
And an example response would be:
Enter the project name: c:/test.pj
After entering the project, it prompts the following:
Do you want to recurse into the subproject test_subprj.pj? [ynYN](n)
At which point I need to respond with either a y or n depending on if I need that subproject. Again, the response to this question is dependent on the subproject. I need to be able to read the subproject in this prompt in order to respond to it with a 'y' or 'n'
Currently I need to manually enter in the project and each of the y's and n's respectively. My goal is to automate this process using Python.
Is there a way to respond to these command line prompts automatically?
Current Progress
Subprocess Strategy
project_path = "c:/test.pj"
with Popen(["si", "viewproject", "--project=" + project_path],
stdin=PIPE, stdout=PIPE, universal_newlines=True) as p:
for line in p.stdout:
if line.startswith("Do you want"):
answer = 'n'
else:
continue # skip it
print(answer, file=p.stdin) # provide answer
p.stdin.flush()
This method is hanging after the with Popen statement. It never errors, but it never enters or exits the for statement and never completes. Currently I am defaulting all answers to "n", but that will be replaced with logic later.
Winpexpect Strategy
import re
import sys
from functools import partial
import winpexpect
project_path = "c:/test.pj"
p = winpexpect.winspawn('si viewproject --project=' + project_path)
p.logfile = sys.stdout
patterns = [re.compile('ynYN'), winpexpect.EOF]
for found in iter(partial(p.expect, patterns), 1): # until EOF
if found == 0:
answer = 'n'
p.sendline(answer)
Returns the following error message:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 541, in read_nonblocking
handle, status, data = self.child_output.get(timeout=timeout)
File "C:\Python33\lib\queue.py", line 175, in get
raise Empty
queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1378, in expect_loop
c = self.read_nonblocking (self.maxread, timeout)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 543, in read_nonblocking
raise TIMEOUT('Timeout exceeded in read_nonblocking().')
pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "K:\eclipse_3.6.0\plugins\org.python.pydev_2.6.0.2012062818\pysrc\pydev_runfiles.py", line 432, in __get_module_from_str
mod = __import__(modname)
File "C:\workspace\Test_prj\Test_prj.py", line 19, in <module>
for found in iter(partial(p.expect, patterns), 1): # until EOF
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1311, in expect
return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1325, in expect_list
return self.expect_loop(searcher_re(pattern_list), timeout, searchwindowsize)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1409, in expect_loop
raise TIMEOUT (str(e) + '\n' + str(self))
pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
<winpexpect.winspawn object at 0x0144AE50>
version: 2.3 ($Revision: 399 $)
command: si
args: ['si', 'viewproject', '--project=c:/test.pj']
searcher: searcher_re:
0: re.compile("ynYN")
1: EOF
buffer (last 100 chars):
before (last 100 chars):
after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 6448
child_fd: 4
closed: False
timeout: 30
delimiter: <class 'pexpect.EOF'>
logfile: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='Cp1252'>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
ERROR: Module: Test_prj could not be imported (file: C:\workspace\Test_prj\Test_prj.py).
Installing Winpexpect
Lazy Persons Way
Install Distribute
Do This
Install PyWin32
Install Winpexpect
Optional: Install Nose
Optional: Install Pip
First-World Problems
Python is a new language for me, and I had never installed a package before for Python. Additionally, Python 3.x is a little different than the other versions of Python making installing modules a little bit more of an adventure.
So, to help others get some sweet sweet module action (and to help those who are more knowledgeable see if I did anything wrong) here's a soon to be success story (hopefully) documenting how I got and installed my first module.
Setup
Python allows third-party groups to develop and distribute modules that extend the abilities of the programming language. Naturally, there is a standard way to help third-party developers make modules as easily available to the end-user as possible.
For Python 3.x, that standard for distributing modules is called Distutils.
Here is how a developer uses Distutils:
Distributing Python Modules
And here is how the end-user uses Distutils:
Installing Python Modules
Normally, navigating to the folder of your downloaded module in the command line and running "setup.py install" will be enough.
BUT
Sometimes life isn't so easy and you may still have problems with your installation. You may, in fact, need something else. For example, you may get the following error:
"ImportError “No Module named Setuptools”"
Luckily, there is a solution for that:
Python 3: ImportError "No Module named Setuptools"
As it turns out, not everything uses distutils. Some packages use setuptools. Unfortunately, there is no setuptools for Python 3.x. Rather, Python 3.x uses distribute which is a branch of setuptools.
So for those who use Python 3.x, here is Distribute: Distribute
For those using Python 2.x, here is Setuptools: Setuptools
In the Installation Instructions for Distribute, it says the following:
"Download
distribute_setup.py <http://python-distribute.org/distribute_setup.py>_
and execute it, using the Python interpreter of your choice."
It also says: "Notice this file is also provided in the source release."
So I downloaded Distribute and saved it to the computer. Once it was saved to the computer, I ran distribute_setup.py from the source release and got the following error:
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.36.tar.gz
Traceback (most recent call last):
File "C:\Python33\lib\urllib\request.py", line 1252, in do_open
h.request(req.get_method(), req.selector, req.data, headers) File "C:\Python33\lib\http\client.py", line 1049, in request
self._send_request(method, url, body, headers)
File "C:\Python33\lib\http\client.py", line 1087, in _send_request
self.endheaders(body)
File "C:\Python33\lib\http\client.py", line 1045, in endheaders
self._send_output(message_body)
File "C:\Python33\lib\http\client.py", line 890, in _send_output
self.send(msg)
File "C:\Python33\lib\http\client.py", line 828, in send
self.connect()
File "C:\Python33\lib\http\client.py", line 806, in connect
self.timeout, self.source_address)
File "C:\Python33\lib\socket.py", line 406, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno 11001] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\workspace\PythonTest\distribute_setup.py", line 553, in <module>
sys.exit(main())
File "C:\workspace\PythonTest\distribute_setup.py", line 549, in main
tarball = download_setuptools(download_base=options.download_base)
File "C:\workspace\PythonTest\distribute_setup.py", line 204, in download_setuptools
src = urlopen(url)
File "C:\Python33\lib\urllib\request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "C:\Python33\lib\urllib\request.py", line 473, in open
response = self._open(req, data)
File "C:\Python33\lib\urllib\request.py", line 491, in _open
'_open', req)
File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain
result = func(*args)
File "C:\Python33\lib\urllib\request.py", line 1272, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Python33\lib\urllib\request.py", line 1255, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>
Well that is no good! I honestly still do not know where that error is coming from or why it happened.
Regardless, then I found the following site that ran a .exe to install distribute as well as pip.
Install Distribute
Install Pip
So I got those installed and then used the following site to setup my computer to more easily use easy_install: Setting Up Easy Install Made Easy
Once I got this working I then installed nose:Nose
The reason I got nose was because the Winpexpect website says:
"WinPexpect includes unit tests. To run the tests, you need nose. Use the following command to run the tests:
$ python setup.py test"
Well that sounds nice :). Now I just wished I knew where to run that test. I know that if you install manually you use the setup.py install command so there will most definitely be a setup.py in the zipped directory online. To see if this was correct, I downloaded and saved the winpexpect file, extracted the information, navigated to it via command-line, and ran setup.py test.
Here was the following result:
running test
running build_py
running egg_info
creating c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info
writing c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\PKG-INFO
writing dependency_links to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\dependency_links.txt
writing top-level names to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\top_level.txt
writing requirements to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\requires.txt
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
reading manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "C:\Documents and Settings\SLZ1FH\Desktop\winpexpect\geertj-winpexpect-76df3cfcb143\setup.py", line 35, in <module>
use_2to3 = True
File "C:\Python33\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Python33\lib\distutils\dist.py", line 917, in run_commands
self.run_command(cmd)
File "C:\Python33\lib\distutils\dist.py", line 936, in run_command
cmd_obj.run()
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 138, in run
self.with_project_on_sys_path(self.run_tests)
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 118, in with_project_on_sys_path
func()
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 164, in run_tests
testLoader = cks
File "C:\Python33\lib\unittest\main.py", line 124, in __init__
self.parseArgs(argv)
File "C:\Python33\lib\unittest\main.py", line 168, in parseArgs
self.createTests()
File "C:\Python33\lib\unittest\main.py", line 175, in createTests
self.module)
File "C:\Python33\lib\unittest\loader.py", line 137, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "C:\Python33\lib\unittest\loader.py", line 137, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "C:\Python33\lib\unittest\loader.py", line 96, in loadTestsFromName
module = __import__('.'.join(parts_copy))
File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\__init__.py", line 1, in <module>
from nose.core import collector, main, run, run_exit, runmodule
File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\core.py", line 143
print "%s version %s" % (os.path.basename(sys.argv[0]), __version__)
^
SyntaxError: invalid syntax
Ok, so the Python 3.3 version of Nose contains invalid syntax for Python 3.3?
print "%s version %s" % (os.path.basename(sys.argv[0]), version)...
should definitely have parenthesis around it... This makes me question if nose will actually work here as it clearly looks to be made for earlier versions of Python.
In the comments you mentioned that xx viewproject < answers.txt > output.txt works but you can't use it because answers depend on the output from the subprocess.
In general pexpect-like modules such as winpexpect (for Windows) could be used. Something like:
import re
import sys
from functools import partial
from winpexpect import EOF, winspawn as spawn
p = spawn('xx viewproject')
p.logfile = sys.stdout
patterns = ['the project:', re.escape('? [ynYN](n)'), EOF]
for found in iter(partial(p.expect, patterns), 2): # until EOF
if found == 0:
p.sendline(project_name)
elif found == 1:
filename = get_filename_from_prompt(p.before) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'no') # a dict
p.sendline(answer)
If the prompts are terminated with a newline (and the subprocess doesn't buffer them); you could read line by line using subprocess module directly:
from subprocess import Popen, PIPE
with Popen(["xx", "viewproject"], stdin=PIPE, stdout=PIPE,
universal_newlines=True) as p:
for line in p.stdout:
if line.startswith("Please enter the name of the project"):
answer = project_name
elif line.startswith("Would you like to recurse into the subproject"):
filename = get_filename_from_prompt(line) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'n') # a dict
else:
continue # skip it
print(answer, file=p.stdin) # provide answer
p.stdin.flush()
To test that you can read something from the xx using subprocess:
from subprocess import Popen, PIPE, STDOUT
with Popen(["xx", "viewproject"], bufsize=0,
stdin=PIPE, stdout=PIPE, stderr=STDOUT) as p:
print(repr(p.stdout.read(1)))
Yes, first of all you may create subprocess as an object by:
p = subprocess.Popen('xx viewproject', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, universal_newlines=True)
Then you'll have methods like communicate() available, for instance:
newline = os.linesep # [1]
commands = ['y', 'n', 'y', 'n', 'y']
p.communicate( newline.join( commands))
1 - os.linesep
Which will send all the answers at once (and hopefully it'll be enough) relying on the same order of question every time.
You may also try parsing p.stdout and then writing to p.stdin, but this may cause deadlock when one buffer will get full while waiting for another, so be careful with this. Luckily there are some complex examples on google.
Simple version would be:
p = Popen(...)
line = p.stdout.readline() # At this point, if child process will wait for stdin
# you have a deadlock on your hands
parse_line( line)
p.stdin.write( newline.join( commands).encode( 'utf-8'))
I would also consider rewriting:
p = subprocess.Popen('si viewproject --project=d:/Projects/test.pj', shell=True,
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
To:
p = subprocess.Popen( ['si', 'viewproject', '--project=d:/Projects/test.pj'],
shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
Unless you explicitly need Shell invocation.
I use python and I want to create a database source controller like liquibase.
I find the python version of liquibase call pyquibase
but get subprocess.CalledProcessError
this is my simple code:
from pyquibase.pyquibase import Pyquibase
if __name__ == '__main__':
pyquibase = Pyquibase.sqlite('test.sqlite', 'db-changelog-1.xml')
pyquibase.update()
and I got these errors:
Traceback (most recent call last):
File "/home/ali/dev/project/python/DatabaseSourceContoller/DatabaseSourceContoller/main.py", line 5, in <module>
pyquibase.update()
File "/home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/pyquibase.py", line 69, in update
output = self.liquibase.execute(self.change_log_file, "update")
File "/home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/liquibase_executor.py", line 103, in execute
shell = True
File "/usr/lib/python3.5/subprocess.py", line 316, in check_output
**kwargs).stdout
File "/usr/lib/python3.5/subprocess.py", line 398, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'java -jar /home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/liquibase/liquibase.jar --driver=org.sqlite.JDBC --classpath=/home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/db-connectors/sqlite-jdbc-3.18.0.jar --changeLogFile=db-changelog-1.xml --url="jdbc:sqlite:test.sqlite" update' returned non-zero exit status 255
pyquibase forks a child process to execute the liquibase changelog update. And the subprocess.CalledProcessError means that the liquibase changelog update has failed.
Now, in order to find out why it failed, you can run the liquibase command manually to see the actual error messages:
java -jar /home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/liquibase/liquibase.jar --driver=org.sqlite.JDBC --classpath=/home/ali/dev/project/python/DatabaseSourceContoller/venv/lib/python3.5/site-packages/pyquibase/db-connectors/sqlite-jdbc-3.18.0.jar --changeLogFile=db-changelog-1.xml --url="jdbc:sqlite:test.sqlite" update
pyquibase doesn't print the actual error messages for you yet. The next version upgrade should have that feature.
I am trying to use Python to interact with another program via the command line. The main problem I am having is a specific call that has multiple follow-up prompts. Initially the command line call asks for the name of a project and then proceeds to ask if I would like to view any of the subfolders of the project. I need to answer y/n to each of these in order and the answer to each is unfortunately not all y or n. Additionally, I cannot know the answer to the question without reading the individual prompts so I am incapable of sending a block of 'y's or 'n's all at once.
This is the command line call:
si viewproject
After entering the command, the command line prompts:
Enter the project name:
And an example response would be:
Enter the project name: c:/test.pj
After entering the project, it prompts the following:
Do you want to recurse into the subproject test_subprj.pj? [ynYN](n)
At which point I need to respond with either a y or n depending on if I need that subproject. Again, the response to this question is dependent on the subproject. I need to be able to read the subproject in this prompt in order to respond to it with a 'y' or 'n'
Currently I need to manually enter in the project and each of the y's and n's respectively. My goal is to automate this process using Python.
Is there a way to respond to these command line prompts automatically?
Current Progress
Subprocess Strategy
project_path = "c:/test.pj"
with Popen(["si", "viewproject", "--project=" + project_path],
stdin=PIPE, stdout=PIPE, universal_newlines=True) as p:
for line in p.stdout:
if line.startswith("Do you want"):
answer = 'n'
else:
continue # skip it
print(answer, file=p.stdin) # provide answer
p.stdin.flush()
This method is hanging after the with Popen statement. It never errors, but it never enters or exits the for statement and never completes. Currently I am defaulting all answers to "n", but that will be replaced with logic later.
Winpexpect Strategy
import re
import sys
from functools import partial
import winpexpect
project_path = "c:/test.pj"
p = winpexpect.winspawn('si viewproject --project=' + project_path)
p.logfile = sys.stdout
patterns = [re.compile('ynYN'), winpexpect.EOF]
for found in iter(partial(p.expect, patterns), 1): # until EOF
if found == 0:
answer = 'n'
p.sendline(answer)
Returns the following error message:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 541, in read_nonblocking
handle, status, data = self.child_output.get(timeout=timeout)
File "C:\Python33\lib\queue.py", line 175, in get
raise Empty
queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1378, in expect_loop
c = self.read_nonblocking (self.maxread, timeout)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 543, in read_nonblocking
raise TIMEOUT('Timeout exceeded in read_nonblocking().')
pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "K:\eclipse_3.6.0\plugins\org.python.pydev_2.6.0.2012062818\pysrc\pydev_runfiles.py", line 432, in __get_module_from_str
mod = __import__(modname)
File "C:\workspace\Test_prj\Test_prj.py", line 19, in <module>
for found in iter(partial(p.expect, patterns), 1): # until EOF
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1311, in expect
return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1325, in expect_list
return self.expect_loop(searcher_re(pattern_list), timeout, searchwindowsize)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1409, in expect_loop
raise TIMEOUT (str(e) + '\n' + str(self))
pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
<winpexpect.winspawn object at 0x0144AE50>
version: 2.3 ($Revision: 399 $)
command: si
args: ['si', 'viewproject', '--project=c:/test.pj']
searcher: searcher_re:
0: re.compile("ynYN")
1: EOF
buffer (last 100 chars):
before (last 100 chars):
after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 6448
child_fd: 4
closed: False
timeout: 30
delimiter: <class 'pexpect.EOF'>
logfile: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='Cp1252'>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
ERROR: Module: Test_prj could not be imported (file: C:\workspace\Test_prj\Test_prj.py).
Installing Winpexpect
Lazy Persons Way
Install Distribute
Do This
Install PyWin32
Install Winpexpect
Optional: Install Nose
Optional: Install Pip
First-World Problems
Python is a new language for me, and I had never installed a package before for Python. Additionally, Python 3.x is a little different than the other versions of Python making installing modules a little bit more of an adventure.
So, to help others get some sweet sweet module action (and to help those who are more knowledgeable see if I did anything wrong) here's a soon to be success story (hopefully) documenting how I got and installed my first module.
Setup
Python allows third-party groups to develop and distribute modules that extend the abilities of the programming language. Naturally, there is a standard way to help third-party developers make modules as easily available to the end-user as possible.
For Python 3.x, that standard for distributing modules is called Distutils.
Here is how a developer uses Distutils:
Distributing Python Modules
And here is how the end-user uses Distutils:
Installing Python Modules
Normally, navigating to the folder of your downloaded module in the command line and running "setup.py install" will be enough.
BUT
Sometimes life isn't so easy and you may still have problems with your installation. You may, in fact, need something else. For example, you may get the following error:
"ImportError “No Module named Setuptools”"
Luckily, there is a solution for that:
Python 3: ImportError "No Module named Setuptools"
As it turns out, not everything uses distutils. Some packages use setuptools. Unfortunately, there is no setuptools for Python 3.x. Rather, Python 3.x uses distribute which is a branch of setuptools.
So for those who use Python 3.x, here is Distribute: Distribute
For those using Python 2.x, here is Setuptools: Setuptools
In the Installation Instructions for Distribute, it says the following:
"Download
distribute_setup.py <http://python-distribute.org/distribute_setup.py>_
and execute it, using the Python interpreter of your choice."
It also says: "Notice this file is also provided in the source release."
So I downloaded Distribute and saved it to the computer. Once it was saved to the computer, I ran distribute_setup.py from the source release and got the following error:
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.36.tar.gz
Traceback (most recent call last):
File "C:\Python33\lib\urllib\request.py", line 1252, in do_open
h.request(req.get_method(), req.selector, req.data, headers) File "C:\Python33\lib\http\client.py", line 1049, in request
self._send_request(method, url, body, headers)
File "C:\Python33\lib\http\client.py", line 1087, in _send_request
self.endheaders(body)
File "C:\Python33\lib\http\client.py", line 1045, in endheaders
self._send_output(message_body)
File "C:\Python33\lib\http\client.py", line 890, in _send_output
self.send(msg)
File "C:\Python33\lib\http\client.py", line 828, in send
self.connect()
File "C:\Python33\lib\http\client.py", line 806, in connect
self.timeout, self.source_address)
File "C:\Python33\lib\socket.py", line 406, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno 11001] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\workspace\PythonTest\distribute_setup.py", line 553, in <module>
sys.exit(main())
File "C:\workspace\PythonTest\distribute_setup.py", line 549, in main
tarball = download_setuptools(download_base=options.download_base)
File "C:\workspace\PythonTest\distribute_setup.py", line 204, in download_setuptools
src = urlopen(url)
File "C:\Python33\lib\urllib\request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "C:\Python33\lib\urllib\request.py", line 473, in open
response = self._open(req, data)
File "C:\Python33\lib\urllib\request.py", line 491, in _open
'_open', req)
File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain
result = func(*args)
File "C:\Python33\lib\urllib\request.py", line 1272, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Python33\lib\urllib\request.py", line 1255, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>
Well that is no good! I honestly still do not know where that error is coming from or why it happened.
Regardless, then I found the following site that ran a .exe to install distribute as well as pip.
Install Distribute
Install Pip
So I got those installed and then used the following site to setup my computer to more easily use easy_install: Setting Up Easy Install Made Easy
Once I got this working I then installed nose:Nose
The reason I got nose was because the Winpexpect website says:
"WinPexpect includes unit tests. To run the tests, you need nose. Use the following command to run the tests:
$ python setup.py test"
Well that sounds nice :). Now I just wished I knew where to run that test. I know that if you install manually you use the setup.py install command so there will most definitely be a setup.py in the zipped directory online. To see if this was correct, I downloaded and saved the winpexpect file, extracted the information, navigated to it via command-line, and ran setup.py test.
Here was the following result:
running test
running build_py
running egg_info
creating c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info
writing c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\PKG-INFO
writing dependency_links to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\dependency_links.txt
writing top-level names to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\top_level.txt
writing requirements to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\requires.txt
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
reading manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "C:\Documents and Settings\SLZ1FH\Desktop\winpexpect\geertj-winpexpect-76df3cfcb143\setup.py", line 35, in <module>
use_2to3 = True
File "C:\Python33\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Python33\lib\distutils\dist.py", line 917, in run_commands
self.run_command(cmd)
File "C:\Python33\lib\distutils\dist.py", line 936, in run_command
cmd_obj.run()
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 138, in run
self.with_project_on_sys_path(self.run_tests)
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 118, in with_project_on_sys_path
func()
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 164, in run_tests
testLoader = cks
File "C:\Python33\lib\unittest\main.py", line 124, in __init__
self.parseArgs(argv)
File "C:\Python33\lib\unittest\main.py", line 168, in parseArgs
self.createTests()
File "C:\Python33\lib\unittest\main.py", line 175, in createTests
self.module)
File "C:\Python33\lib\unittest\loader.py", line 137, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "C:\Python33\lib\unittest\loader.py", line 137, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "C:\Python33\lib\unittest\loader.py", line 96, in loadTestsFromName
module = __import__('.'.join(parts_copy))
File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\__init__.py", line 1, in <module>
from nose.core import collector, main, run, run_exit, runmodule
File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\core.py", line 143
print "%s version %s" % (os.path.basename(sys.argv[0]), __version__)
^
SyntaxError: invalid syntax
Ok, so the Python 3.3 version of Nose contains invalid syntax for Python 3.3?
print "%s version %s" % (os.path.basename(sys.argv[0]), version)...
should definitely have parenthesis around it... This makes me question if nose will actually work here as it clearly looks to be made for earlier versions of Python.
In the comments you mentioned that xx viewproject < answers.txt > output.txt works but you can't use it because answers depend on the output from the subprocess.
In general pexpect-like modules such as winpexpect (for Windows) could be used. Something like:
import re
import sys
from functools import partial
from winpexpect import EOF, winspawn as spawn
p = spawn('xx viewproject')
p.logfile = sys.stdout
patterns = ['the project:', re.escape('? [ynYN](n)'), EOF]
for found in iter(partial(p.expect, patterns), 2): # until EOF
if found == 0:
p.sendline(project_name)
elif found == 1:
filename = get_filename_from_prompt(p.before) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'no') # a dict
p.sendline(answer)
If the prompts are terminated with a newline (and the subprocess doesn't buffer them); you could read line by line using subprocess module directly:
from subprocess import Popen, PIPE
with Popen(["xx", "viewproject"], stdin=PIPE, stdout=PIPE,
universal_newlines=True) as p:
for line in p.stdout:
if line.startswith("Please enter the name of the project"):
answer = project_name
elif line.startswith("Would you like to recurse into the subproject"):
filename = get_filename_from_prompt(line) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'n') # a dict
else:
continue # skip it
print(answer, file=p.stdin) # provide answer
p.stdin.flush()
To test that you can read something from the xx using subprocess:
from subprocess import Popen, PIPE, STDOUT
with Popen(["xx", "viewproject"], bufsize=0,
stdin=PIPE, stdout=PIPE, stderr=STDOUT) as p:
print(repr(p.stdout.read(1)))
Yes, first of all you may create subprocess as an object by:
p = subprocess.Popen('xx viewproject', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, universal_newlines=True)
Then you'll have methods like communicate() available, for instance:
newline = os.linesep # [1]
commands = ['y', 'n', 'y', 'n', 'y']
p.communicate( newline.join( commands))
1 - os.linesep
Which will send all the answers at once (and hopefully it'll be enough) relying on the same order of question every time.
You may also try parsing p.stdout and then writing to p.stdin, but this may cause deadlock when one buffer will get full while waiting for another, so be careful with this. Luckily there are some complex examples on google.
Simple version would be:
p = Popen(...)
line = p.stdout.readline() # At this point, if child process will wait for stdin
# you have a deadlock on your hands
parse_line( line)
p.stdin.write( newline.join( commands).encode( 'utf-8'))
I would also consider rewriting:
p = subprocess.Popen('si viewproject --project=d:/Projects/test.pj', shell=True,
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
To:
p = subprocess.Popen( ['si', 'viewproject', '--project=d:/Projects/test.pj'],
shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
Unless you explicitly need Shell invocation.
When I try to execute a python program from command line, it gives the following error. These errors do not cause any problem to my ouput. I dont want it to be displayed in the commandline
Traceback (most recent call last):
File "test.py", line 88, in <module>
p.feed(ht)
File "/usr/lib/python2.5/HTMLParser.py", line 108, in feed
self.goahead(0)
File "/usr/lib/python2.5/HTMLParser.py", line 148, in goahead
k = self.parse_starttag(i)
File "/usr/lib/python2.5/HTMLParser.py", line 226, in parse_starttag
endpos = self.check_for_whole_start_tag(i)
File "/usr/lib/python2.5/HTMLParser.py", line 301, in check_for_whole_start_tag
self.error("malformed start tag")
File "/usr/lib/python2.5/HTMLParser.py", line 115, in error
raise HTMLParseError(message, self.getpos())
HTMLParser.HTMLParseError: malformed start tag, at line 319, column 25
How could I suppress the errors?
Doesn't catching HTMLParseError work for you? If test.py is the name of your python file, it's propagated up to there, so it should.
Here's an example how to suppress such an error. You might want to tweak it a bit to match your code.
try:
# Put parsing code here
except HTMLParseError:
pass
You can also just suppress the error message by redirecting stderr to null, like Ignacio suggested. To do it in code, you can just write the following:
import sys
class DevNull:
def write(self, msg):
pass
sys.stderr = DevNull()
However, this is probably not be what you want, because from your error it looks like the script execution is stopped, and you probably want it to be continued.
Redirect stderr to /dev/null.
python somescript.py 2> /dev/null
In python 3, #Boaz Yaniv's answer can be simplified as
sys.stderr = object
since every class in python3 is inherited from Object, so technically this would work, at least I've tried it by myself in python 3.6.5 environment.
Here is a more readable, succinct solution for handling errors that are safe to ignore, without having to resort to the typical try/except/pass code block.
from contextlib import suppress
with suppress(IgnorableErrorA, IgnorableErrorB):
do_something()