I am trying to use Python to interact with another program via the command line. The main problem I am having is a specific call that has multiple follow-up prompts. Initially the command line call asks for the name of a project and then proceeds to ask if I would like to view any of the subfolders of the project. I need to answer y/n to each of these in order and the answer to each is unfortunately not all y or n. Additionally, I cannot know the answer to the question without reading the individual prompts so I am incapable of sending a block of 'y's or 'n's all at once.
This is the command line call:
si viewproject
After entering the command, the command line prompts:
Enter the project name:
And an example response would be:
Enter the project name: c:/test.pj
After entering the project, it prompts the following:
Do you want to recurse into the subproject test_subprj.pj? [ynYN](n)
At which point I need to respond with either a y or n depending on if I need that subproject. Again, the response to this question is dependent on the subproject. I need to be able to read the subproject in this prompt in order to respond to it with a 'y' or 'n'
Currently I need to manually enter in the project and each of the y's and n's respectively. My goal is to automate this process using Python.
Is there a way to respond to these command line prompts automatically?
Current Progress
Subprocess Strategy
project_path = "c:/test.pj"
with Popen(["si", "viewproject", "--project=" + project_path],
stdin=PIPE, stdout=PIPE, universal_newlines=True) as p:
for line in p.stdout:
if line.startswith("Do you want"):
answer = 'n'
else:
continue # skip it
print(answer, file=p.stdin) # provide answer
p.stdin.flush()
This method is hanging after the with Popen statement. It never errors, but it never enters or exits the for statement and never completes. Currently I am defaulting all answers to "n", but that will be replaced with logic later.
Winpexpect Strategy
import re
import sys
from functools import partial
import winpexpect
project_path = "c:/test.pj"
p = winpexpect.winspawn('si viewproject --project=' + project_path)
p.logfile = sys.stdout
patterns = [re.compile('ynYN'), winpexpect.EOF]
for found in iter(partial(p.expect, patterns), 1): # until EOF
if found == 0:
answer = 'n'
p.sendline(answer)
Returns the following error message:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 541, in read_nonblocking
handle, status, data = self.child_output.get(timeout=timeout)
File "C:\Python33\lib\queue.py", line 175, in get
raise Empty
queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1378, in expect_loop
c = self.read_nonblocking (self.maxread, timeout)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 543, in read_nonblocking
raise TIMEOUT('Timeout exceeded in read_nonblocking().')
pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "K:\eclipse_3.6.0\plugins\org.python.pydev_2.6.0.2012062818\pysrc\pydev_runfiles.py", line 432, in __get_module_from_str
mod = __import__(modname)
File "C:\workspace\Test_prj\Test_prj.py", line 19, in <module>
for found in iter(partial(p.expect, patterns), 1): # until EOF
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1311, in expect
return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1325, in expect_list
return self.expect_loop(searcher_re(pattern_list), timeout, searchwindowsize)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1409, in expect_loop
raise TIMEOUT (str(e) + '\n' + str(self))
pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
<winpexpect.winspawn object at 0x0144AE50>
version: 2.3 ($Revision: 399 $)
command: si
args: ['si', 'viewproject', '--project=c:/test.pj']
searcher: searcher_re:
0: re.compile("ynYN")
1: EOF
buffer (last 100 chars):
before (last 100 chars):
after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 6448
child_fd: 4
closed: False
timeout: 30
delimiter: <class 'pexpect.EOF'>
logfile: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='Cp1252'>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
ERROR: Module: Test_prj could not be imported (file: C:\workspace\Test_prj\Test_prj.py).
Installing Winpexpect
Lazy Persons Way
Install Distribute
Do This
Install PyWin32
Install Winpexpect
Optional: Install Nose
Optional: Install Pip
First-World Problems
Python is a new language for me, and I had never installed a package before for Python. Additionally, Python 3.x is a little different than the other versions of Python making installing modules a little bit more of an adventure.
So, to help others get some sweet sweet module action (and to help those who are more knowledgeable see if I did anything wrong) here's a soon to be success story (hopefully) documenting how I got and installed my first module.
Setup
Python allows third-party groups to develop and distribute modules that extend the abilities of the programming language. Naturally, there is a standard way to help third-party developers make modules as easily available to the end-user as possible.
For Python 3.x, that standard for distributing modules is called Distutils.
Here is how a developer uses Distutils:
Distributing Python Modules
And here is how the end-user uses Distutils:
Installing Python Modules
Normally, navigating to the folder of your downloaded module in the command line and running "setup.py install" will be enough.
BUT
Sometimes life isn't so easy and you may still have problems with your installation. You may, in fact, need something else. For example, you may get the following error:
"ImportError “No Module named Setuptools”"
Luckily, there is a solution for that:
Python 3: ImportError "No Module named Setuptools"
As it turns out, not everything uses distutils. Some packages use setuptools. Unfortunately, there is no setuptools for Python 3.x. Rather, Python 3.x uses distribute which is a branch of setuptools.
So for those who use Python 3.x, here is Distribute: Distribute
For those using Python 2.x, here is Setuptools: Setuptools
In the Installation Instructions for Distribute, it says the following:
"Download
distribute_setup.py <http://python-distribute.org/distribute_setup.py>_
and execute it, using the Python interpreter of your choice."
It also says: "Notice this file is also provided in the source release."
So I downloaded Distribute and saved it to the computer. Once it was saved to the computer, I ran distribute_setup.py from the source release and got the following error:
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.36.tar.gz
Traceback (most recent call last):
File "C:\Python33\lib\urllib\request.py", line 1252, in do_open
h.request(req.get_method(), req.selector, req.data, headers) File "C:\Python33\lib\http\client.py", line 1049, in request
self._send_request(method, url, body, headers)
File "C:\Python33\lib\http\client.py", line 1087, in _send_request
self.endheaders(body)
File "C:\Python33\lib\http\client.py", line 1045, in endheaders
self._send_output(message_body)
File "C:\Python33\lib\http\client.py", line 890, in _send_output
self.send(msg)
File "C:\Python33\lib\http\client.py", line 828, in send
self.connect()
File "C:\Python33\lib\http\client.py", line 806, in connect
self.timeout, self.source_address)
File "C:\Python33\lib\socket.py", line 406, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno 11001] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\workspace\PythonTest\distribute_setup.py", line 553, in <module>
sys.exit(main())
File "C:\workspace\PythonTest\distribute_setup.py", line 549, in main
tarball = download_setuptools(download_base=options.download_base)
File "C:\workspace\PythonTest\distribute_setup.py", line 204, in download_setuptools
src = urlopen(url)
File "C:\Python33\lib\urllib\request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "C:\Python33\lib\urllib\request.py", line 473, in open
response = self._open(req, data)
File "C:\Python33\lib\urllib\request.py", line 491, in _open
'_open', req)
File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain
result = func(*args)
File "C:\Python33\lib\urllib\request.py", line 1272, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Python33\lib\urllib\request.py", line 1255, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>
Well that is no good! I honestly still do not know where that error is coming from or why it happened.
Regardless, then I found the following site that ran a .exe to install distribute as well as pip.
Install Distribute
Install Pip
So I got those installed and then used the following site to setup my computer to more easily use easy_install: Setting Up Easy Install Made Easy
Once I got this working I then installed nose:Nose
The reason I got nose was because the Winpexpect website says:
"WinPexpect includes unit tests. To run the tests, you need nose. Use the following command to run the tests:
$ python setup.py test"
Well that sounds nice :). Now I just wished I knew where to run that test. I know that if you install manually you use the setup.py install command so there will most definitely be a setup.py in the zipped directory online. To see if this was correct, I downloaded and saved the winpexpect file, extracted the information, navigated to it via command-line, and ran setup.py test.
Here was the following result:
running test
running build_py
running egg_info
creating c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info
writing c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\PKG-INFO
writing dependency_links to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\dependency_links.txt
writing top-level names to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\top_level.txt
writing requirements to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\requires.txt
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
reading manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "C:\Documents and Settings\SLZ1FH\Desktop\winpexpect\geertj-winpexpect-76df3cfcb143\setup.py", line 35, in <module>
use_2to3 = True
File "C:\Python33\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Python33\lib\distutils\dist.py", line 917, in run_commands
self.run_command(cmd)
File "C:\Python33\lib\distutils\dist.py", line 936, in run_command
cmd_obj.run()
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 138, in run
self.with_project_on_sys_path(self.run_tests)
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 118, in with_project_on_sys_path
func()
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 164, in run_tests
testLoader = cks
File "C:\Python33\lib\unittest\main.py", line 124, in __init__
self.parseArgs(argv)
File "C:\Python33\lib\unittest\main.py", line 168, in parseArgs
self.createTests()
File "C:\Python33\lib\unittest\main.py", line 175, in createTests
self.module)
File "C:\Python33\lib\unittest\loader.py", line 137, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "C:\Python33\lib\unittest\loader.py", line 137, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "C:\Python33\lib\unittest\loader.py", line 96, in loadTestsFromName
module = __import__('.'.join(parts_copy))
File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\__init__.py", line 1, in <module>
from nose.core import collector, main, run, run_exit, runmodule
File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\core.py", line 143
print "%s version %s" % (os.path.basename(sys.argv[0]), __version__)
^
SyntaxError: invalid syntax
Ok, so the Python 3.3 version of Nose contains invalid syntax for Python 3.3?
print "%s version %s" % (os.path.basename(sys.argv[0]), version)...
should definitely have parenthesis around it... This makes me question if nose will actually work here as it clearly looks to be made for earlier versions of Python.
In the comments you mentioned that xx viewproject < answers.txt > output.txt works but you can't use it because answers depend on the output from the subprocess.
In general pexpect-like modules such as winpexpect (for Windows) could be used. Something like:
import re
import sys
from functools import partial
from winpexpect import EOF, winspawn as spawn
p = spawn('xx viewproject')
p.logfile = sys.stdout
patterns = ['the project:', re.escape('? [ynYN](n)'), EOF]
for found in iter(partial(p.expect, patterns), 2): # until EOF
if found == 0:
p.sendline(project_name)
elif found == 1:
filename = get_filename_from_prompt(p.before) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'no') # a dict
p.sendline(answer)
If the prompts are terminated with a newline (and the subprocess doesn't buffer them); you could read line by line using subprocess module directly:
from subprocess import Popen, PIPE
with Popen(["xx", "viewproject"], stdin=PIPE, stdout=PIPE,
universal_newlines=True) as p:
for line in p.stdout:
if line.startswith("Please enter the name of the project"):
answer = project_name
elif line.startswith("Would you like to recurse into the subproject"):
filename = get_filename_from_prompt(line) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'n') # a dict
else:
continue # skip it
print(answer, file=p.stdin) # provide answer
p.stdin.flush()
To test that you can read something from the xx using subprocess:
from subprocess import Popen, PIPE, STDOUT
with Popen(["xx", "viewproject"], bufsize=0,
stdin=PIPE, stdout=PIPE, stderr=STDOUT) as p:
print(repr(p.stdout.read(1)))
Yes, first of all you may create subprocess as an object by:
p = subprocess.Popen('xx viewproject', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, universal_newlines=True)
Then you'll have methods like communicate() available, for instance:
newline = os.linesep # [1]
commands = ['y', 'n', 'y', 'n', 'y']
p.communicate( newline.join( commands))
1 - os.linesep
Which will send all the answers at once (and hopefully it'll be enough) relying on the same order of question every time.
You may also try parsing p.stdout and then writing to p.stdin, but this may cause deadlock when one buffer will get full while waiting for another, so be careful with this. Luckily there are some complex examples on google.
Simple version would be:
p = Popen(...)
line = p.stdout.readline() # At this point, if child process will wait for stdin
# you have a deadlock on your hands
parse_line( line)
p.stdin.write( newline.join( commands).encode( 'utf-8'))
I would also consider rewriting:
p = subprocess.Popen('si viewproject --project=d:/Projects/test.pj', shell=True,
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
To:
p = subprocess.Popen( ['si', 'viewproject', '--project=d:/Projects/test.pj'],
shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
Unless you explicitly need Shell invocation.
Related
I was trying Streamlit 1.13 on Windows 10, where I encountered the following error:
Z:\>streamlit run st1.py
2022-10-04 02:25:28.218 INFO numexpr.utils: NumExpr defaulting to 4 threads.
Welcome to Streamlit!
If you're one of our development partners or you're interested in getting
personal technical support or Streamlit updates, please enter your email
address below. Otherwise, you may leave the field blank.
http://localhost:8501
Traceback (most recent call last):
File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\tornado\http1connection.py", line 276, in _read_message
delegate.finish()
File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\tornado\routing.py", line 268, in finish
self.delegate.finish()
File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\tornado\web.py", line 2322, in finish
self.execute()
File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\tornado\web.py", line 2344, in execute
self.handler = self.handler_class(
File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\tornado\web.py", line 239, in __init__
self.initialize(**kwargs) # type: ignore
File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\streamlit\web\server\routes.py", line 49, in initialize
self._pages = get_pages()
File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\streamlit\web\server\server.py", line 397, in <lambda>
for page_info in source_util.get_pages(
File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\streamlit\source_util.py", line 155, in get_pages
"script_path": str(main_script_path.resolve()),
File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\pathlib.py", line 1215, in resolve
s = self._flavour.resolve(self, strict=strict)
File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\pathlib.py", line 215, in resolve
s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 1] Incorrect function: 'st1.py'
The installation of streamlit was complete: initially there was a conflict which I fixed, I also installed it in Anaconda and the error was the same.
I checked the exact streamlit file which rised the exception and changed the script to print the actual path of the script and it was correct, as well as the file was there.
#File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\streamlit\source_util.py", line 155, in get_pages
def get_pages(main_script_path_str: str) -> Dict[str, Dict[str, str]]:
global _cached_pages
print("main_script_path_str=",main_script_path_str) #DEBUG
# Avoid taking the lock if the pages cache hasn't been invalidated.
pages = _cached_pages
if pages is not None:
return pages
The problem was with the location of the script on a RAM disk. Moving it to the a regular disk resolved it.
Another question with this Win error reminded me that the drivers of Windows RAM drives, at least the ones I've used such as Imdisk or OSFMount, seem to have missing support of some of the OS file functions.
OSError: [WinError 1] Incorrect function
E.g. "Rust" also had errors when trying to build source located in any of these RAM drives on Windows.
I'm having trouble installing things without error when using Pipenv and am unable to figure out how to fix the error. Just for some upfront clarification, I'm using Windows 10, Python 3.9 and the issue is when I install with Pipenv. My last name is a little odd and has an apostrophe in it; however, I usually leave the apostrophe out when registering for things due to potential issues with the apostrophe. For some reason, at work, the IT department created my account using the apostrophe. Since it is configured with Azure AD, I cannot adjust the username from my own PC, even though I have admin rights on this machine.
The error seems to be coming from my opening a JSON file in my appdata directory, which , when fully qualified contains the string "t'alua" (my last name). Unfortunately, the string seems to be enclosed in single quotes so the apostrophe in my name causes a parsing issue. Just for the record, I have even uninstalled python from my C drive and installed on my D drive and executed commands in my D drive (where there is no apostrophe in the working directory's path) but I still get the same issue since it is related to the appdata directory's absolute path. I have also asked the IT department to change my username to remove the apostrophe but they said its not feasible due to so many downstream changes that would be required to prevent any kind of bugs, which I understand.
Here is an example command with the error shown:
(corl_anl_coercion-sJyNKu6O) D:\mttf_mtbf_datasets\training_data\corl_anl_coercion>pipenv install numpy
Installing numpy...
[= ] Installing numpy...Failed to load paths: File "<string>", line 1
import sysconfig, distutils.sysconfig, io, json, sys; paths = {u'purelib': u'{0}'.format(distutils.sysconfig.get_python_lib(plat_specific=0)),u'stdlib': u'{0}'.format(sysconfig.get_path('stdlib')),u'platlib': u'{0}'.format(distutils.sysconfig.get_python_lib(plat_specific=1)),u'platstdlib': u'{0}'.format(sysconfig.get_path('platstdlib')),u'include': u'{0}'.format(distutils.sysconfig.get_python_inc(plat_specific=0)),u'platinclude': u'{0}'.format(distutils.sysconfig.get_python_inc(plat_specific=1)),u'scripts': u'{0}'.format(sysconfig.get_path('scripts')),u'py_version_short': u'{0}'.format(distutils.sysconfig.get_python_version()), }; value = u'{0}'.format(json.dumps(paths));fh = io.open('c:/users/t'alua~1/appdata/local/temp/tmp0m5ovbrb.json', 'w'); fh.write(value); fh.close()
^
SyntaxError: invalid syntax
Output:
[ ==] Installing numpy...Failed to load paths: File "<string>", line 1
import sysconfig, distutils.sysconfig, io, json, sys; paths = {u'purelib': u'{0}'.format(distutils.sysconfig.get_python_lib(plat_specific=0)),u'stdlib': u'{0}'.format(sysconfig.get_path('stdlib')),u'platlib': u'{0}'.format(distutils.sysconfig.get_python_lib(plat_specific=1)),u'platstdlib': u'{0}'.format(sysconfig.get_path('platstdlib')),u'include': u'{0}'.format(distutils.sysconfig.get_python_inc(plat_specific=0)),u'platinclude': u'{0}'.format(distutils.sysconfig.get_python_inc(plat_specific=1)),u'scripts': u'{0}'.format(sysconfig.get_path('scripts')),u'py_version_short': u'{0}'.format(distutils.sysconfig.get_python_version()), }; value = u'{0}'.format(json.dumps(paths));fh = io.open('c:/users/t'alua~1/appdata/local/temp/tmpywegc7gn.json', 'w'); fh.write(value); fh.close()
^
SyntaxError: invalid syntax
Output:
Adding numpy to Pipfile's [packages]...
Installation Succeeded
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Resolving dependencies...
Locking Failed!
Traceback (most recent call last):
File "D:\installations\python\lib\site-packages\pipenv\resolver.py", line 764, in <module>
main()
File "D:\installations\python\lib\site-packages\pipenv\resolver.py", line 758, in main
_main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
File "D:\installations\python\lib\site-packages\pipenv\resolver.py", line 741, in _main
resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
File "D:\installations\python\lib\site-packages\pipenv\resolver.py", line 702, in resolve_packages
results, resolver = resolve(
File "D:\installations\python\lib\site-packages\pipenv\resolver.py", line 684, in resolve
return resolve_deps(
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 1406, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 1119, in actually_resolve_deps
resolver.resolve()
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 834, in resolve
results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 822, in resolver
self.get_resolver(clear=self.clear, pre=self.pre)
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 813, in get_resolver
constraints=self.parsed_constraints, repository=self.repository,
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 806, in parsed_constraints
self._parsed_constraints = [c for c in self.constraints]
File "d:\installations\python\lib\site-packages\pipenv\utils.py", line 806, in <listcomp>
self._parsed_constraints = [c for c in self.constraints]
File "d:\installations\python\lib\site-packages\pipenv\patched\notpip\_internal\req\req_file.py", line 137, in parse_requirements
for parsed_line in parser.parse(filename, constraint):
File "d:\installations\python\lib\site-packages\pipenv\patched\notpip\_internal\req\req_file.py", line 282, in parse
for line in self._parse_and_recurse(filename, constraint):
File "d:\installations\python\lib\site-packages\pipenv\patched\notpip\_internal\req\req_file.py", line 287, in _parse_and_recurse
for line in self._parse_file(filename, constraint):
File "d:\installations\python\lib\site-packages\pipenv\patched\notpip\_internal\req\req_file.py", line 329, in _parse_file
args_str, opts = self._line_parser(line)
File "d:\installations\python\lib\site-packages\pipenv\patched\notpip\_internal\req\req_file.py", line 365, in parse_line
shlex.split(options_str), defaults) # type: ignore
File "D:\installations\python\lib\shlex.py", line 315, in split
return list(lex)
File "D:\installations\python\lib\shlex.py", line 300, in __next__
token = self.get_token()
File "D:\installations\python\lib\shlex.py", line 109, in get_token
raw = self.read_token()
File "D:\installations\python\lib\shlex.py", line 191, in read_token
raise ValueError("No closing quotation")
ValueError: No closing quotation
I can see from the output above that the error is due to this line of python being executed:
fh = io.open('c:/users/t'alua~1/appdata/local/temp/tmp0m5ovbrb.json', 'w');
since the string is not terminated correctly.
Unfortunately, I'm not sure where this is coming from, though. I tried to read through each of the python files listed in the stack trace and I see where, in shlex.py if the string ends, while still being in a 'quoted state', then it raises the "No closing Quotation" valueerror. I couldn't see any line of code where it actually pulls the appdata path or why it is using appdata in the first place. I did find in my python installation's site packages directory's util.py:
D:\installations\python\Lib\site-packages\pipenv\utils.py
that there is an escaped_grouped_arguments function. I thought perhaps if the filepath is being parsed with this, I can modify the function to check for variations of my lastname and replace it with a good value so I made the following change:
def escape_grouped_arguments(s):
"""Prepares a string for the shell (on Windows too!)
Only for use on grouped arguments (passed as a string to Popen)
"""
if s is None:
return None
## This is the original code
# Additional escaping for windows paths
#if os.name == "nt":
# s = "{}".format(s.replace("\\", "\\\\"))
#return '"' + s.replace("'", "'\\''") + '"'
## This is my modification
if "t'alua" in s.lower():
bad_names = ["T'Alua","t'alua","T'alua","T'ALUA"]
good_name = "talua"
for bad_name in bad_names:
s = s.replace(bad_name, good_name)
if os.name == "nt":
s = "{}".format(s.replace("\\", "\\\\"))
return '"' + s.replace("'", "'\\''") + '"'
But this change didn't seem to resolve the errors. Any and all guidance on how to correctly spot exactly where the error is coming from as well as how to best fix this particular issue would be greatly appreciated.
I am trying to use Python to interact with another program via the command line. The main problem I am having is a specific call that has multiple follow-up prompts. Initially the command line call asks for the name of a project and then proceeds to ask if I would like to view any of the subfolders of the project. I need to answer y/n to each of these in order and the answer to each is unfortunately not all y or n. Additionally, I cannot know the answer to the question without reading the individual prompts so I am incapable of sending a block of 'y's or 'n's all at once.
This is the command line call:
si viewproject
After entering the command, the command line prompts:
Enter the project name:
And an example response would be:
Enter the project name: c:/test.pj
After entering the project, it prompts the following:
Do you want to recurse into the subproject test_subprj.pj? [ynYN](n)
At which point I need to respond with either a y or n depending on if I need that subproject. Again, the response to this question is dependent on the subproject. I need to be able to read the subproject in this prompt in order to respond to it with a 'y' or 'n'
Currently I need to manually enter in the project and each of the y's and n's respectively. My goal is to automate this process using Python.
Is there a way to respond to these command line prompts automatically?
Current Progress
Subprocess Strategy
project_path = "c:/test.pj"
with Popen(["si", "viewproject", "--project=" + project_path],
stdin=PIPE, stdout=PIPE, universal_newlines=True) as p:
for line in p.stdout:
if line.startswith("Do you want"):
answer = 'n'
else:
continue # skip it
print(answer, file=p.stdin) # provide answer
p.stdin.flush()
This method is hanging after the with Popen statement. It never errors, but it never enters or exits the for statement and never completes. Currently I am defaulting all answers to "n", but that will be replaced with logic later.
Winpexpect Strategy
import re
import sys
from functools import partial
import winpexpect
project_path = "c:/test.pj"
p = winpexpect.winspawn('si viewproject --project=' + project_path)
p.logfile = sys.stdout
patterns = [re.compile('ynYN'), winpexpect.EOF]
for found in iter(partial(p.expect, patterns), 1): # until EOF
if found == 0:
answer = 'n'
p.sendline(answer)
Returns the following error message:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 541, in read_nonblocking
handle, status, data = self.child_output.get(timeout=timeout)
File "C:\Python33\lib\queue.py", line 175, in get
raise Empty
queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1378, in expect_loop
c = self.read_nonblocking (self.maxread, timeout)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 543, in read_nonblocking
raise TIMEOUT('Timeout exceeded in read_nonblocking().')
pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "K:\eclipse_3.6.0\plugins\org.python.pydev_2.6.0.2012062818\pysrc\pydev_runfiles.py", line 432, in __get_module_from_str
mod = __import__(modname)
File "C:\workspace\Test_prj\Test_prj.py", line 19, in <module>
for found in iter(partial(p.expect, patterns), 1): # until EOF
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1311, in expect
return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1325, in expect_list
return self.expect_loop(searcher_re(pattern_list), timeout, searchwindowsize)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1409, in expect_loop
raise TIMEOUT (str(e) + '\n' + str(self))
pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
<winpexpect.winspawn object at 0x0144AE50>
version: 2.3 ($Revision: 399 $)
command: si
args: ['si', 'viewproject', '--project=c:/test.pj']
searcher: searcher_re:
0: re.compile("ynYN")
1: EOF
buffer (last 100 chars):
before (last 100 chars):
after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 6448
child_fd: 4
closed: False
timeout: 30
delimiter: <class 'pexpect.EOF'>
logfile: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='Cp1252'>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
ERROR: Module: Test_prj could not be imported (file: C:\workspace\Test_prj\Test_prj.py).
Installing Winpexpect
Lazy Persons Way
Install Distribute
Do This
Install PyWin32
Install Winpexpect
Optional: Install Nose
Optional: Install Pip
First-World Problems
Python is a new language for me, and I had never installed a package before for Python. Additionally, Python 3.x is a little different than the other versions of Python making installing modules a little bit more of an adventure.
So, to help others get some sweet sweet module action (and to help those who are more knowledgeable see if I did anything wrong) here's a soon to be success story (hopefully) documenting how I got and installed my first module.
Setup
Python allows third-party groups to develop and distribute modules that extend the abilities of the programming language. Naturally, there is a standard way to help third-party developers make modules as easily available to the end-user as possible.
For Python 3.x, that standard for distributing modules is called Distutils.
Here is how a developer uses Distutils:
Distributing Python Modules
And here is how the end-user uses Distutils:
Installing Python Modules
Normally, navigating to the folder of your downloaded module in the command line and running "setup.py install" will be enough.
BUT
Sometimes life isn't so easy and you may still have problems with your installation. You may, in fact, need something else. For example, you may get the following error:
"ImportError “No Module named Setuptools”"
Luckily, there is a solution for that:
Python 3: ImportError "No Module named Setuptools"
As it turns out, not everything uses distutils. Some packages use setuptools. Unfortunately, there is no setuptools for Python 3.x. Rather, Python 3.x uses distribute which is a branch of setuptools.
So for those who use Python 3.x, here is Distribute: Distribute
For those using Python 2.x, here is Setuptools: Setuptools
In the Installation Instructions for Distribute, it says the following:
"Download
distribute_setup.py <http://python-distribute.org/distribute_setup.py>_
and execute it, using the Python interpreter of your choice."
It also says: "Notice this file is also provided in the source release."
So I downloaded Distribute and saved it to the computer. Once it was saved to the computer, I ran distribute_setup.py from the source release and got the following error:
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.36.tar.gz
Traceback (most recent call last):
File "C:\Python33\lib\urllib\request.py", line 1252, in do_open
h.request(req.get_method(), req.selector, req.data, headers) File "C:\Python33\lib\http\client.py", line 1049, in request
self._send_request(method, url, body, headers)
File "C:\Python33\lib\http\client.py", line 1087, in _send_request
self.endheaders(body)
File "C:\Python33\lib\http\client.py", line 1045, in endheaders
self._send_output(message_body)
File "C:\Python33\lib\http\client.py", line 890, in _send_output
self.send(msg)
File "C:\Python33\lib\http\client.py", line 828, in send
self.connect()
File "C:\Python33\lib\http\client.py", line 806, in connect
self.timeout, self.source_address)
File "C:\Python33\lib\socket.py", line 406, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno 11001] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\workspace\PythonTest\distribute_setup.py", line 553, in <module>
sys.exit(main())
File "C:\workspace\PythonTest\distribute_setup.py", line 549, in main
tarball = download_setuptools(download_base=options.download_base)
File "C:\workspace\PythonTest\distribute_setup.py", line 204, in download_setuptools
src = urlopen(url)
File "C:\Python33\lib\urllib\request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "C:\Python33\lib\urllib\request.py", line 473, in open
response = self._open(req, data)
File "C:\Python33\lib\urllib\request.py", line 491, in _open
'_open', req)
File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain
result = func(*args)
File "C:\Python33\lib\urllib\request.py", line 1272, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Python33\lib\urllib\request.py", line 1255, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>
Well that is no good! I honestly still do not know where that error is coming from or why it happened.
Regardless, then I found the following site that ran a .exe to install distribute as well as pip.
Install Distribute
Install Pip
So I got those installed and then used the following site to setup my computer to more easily use easy_install: Setting Up Easy Install Made Easy
Once I got this working I then installed nose:Nose
The reason I got nose was because the Winpexpect website says:
"WinPexpect includes unit tests. To run the tests, you need nose. Use the following command to run the tests:
$ python setup.py test"
Well that sounds nice :). Now I just wished I knew where to run that test. I know that if you install manually you use the setup.py install command so there will most definitely be a setup.py in the zipped directory online. To see if this was correct, I downloaded and saved the winpexpect file, extracted the information, navigated to it via command-line, and ran setup.py test.
Here was the following result:
running test
running build_py
running egg_info
creating c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info
writing c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\PKG-INFO
writing dependency_links to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\dependency_links.txt
writing top-level names to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\top_level.txt
writing requirements to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\requires.txt
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
reading manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "C:\Documents and Settings\SLZ1FH\Desktop\winpexpect\geertj-winpexpect-76df3cfcb143\setup.py", line 35, in <module>
use_2to3 = True
File "C:\Python33\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Python33\lib\distutils\dist.py", line 917, in run_commands
self.run_command(cmd)
File "C:\Python33\lib\distutils\dist.py", line 936, in run_command
cmd_obj.run()
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 138, in run
self.with_project_on_sys_path(self.run_tests)
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 118, in with_project_on_sys_path
func()
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 164, in run_tests
testLoader = cks
File "C:\Python33\lib\unittest\main.py", line 124, in __init__
self.parseArgs(argv)
File "C:\Python33\lib\unittest\main.py", line 168, in parseArgs
self.createTests()
File "C:\Python33\lib\unittest\main.py", line 175, in createTests
self.module)
File "C:\Python33\lib\unittest\loader.py", line 137, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "C:\Python33\lib\unittest\loader.py", line 137, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "C:\Python33\lib\unittest\loader.py", line 96, in loadTestsFromName
module = __import__('.'.join(parts_copy))
File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\__init__.py", line 1, in <module>
from nose.core import collector, main, run, run_exit, runmodule
File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\core.py", line 143
print "%s version %s" % (os.path.basename(sys.argv[0]), __version__)
^
SyntaxError: invalid syntax
Ok, so the Python 3.3 version of Nose contains invalid syntax for Python 3.3?
print "%s version %s" % (os.path.basename(sys.argv[0]), version)...
should definitely have parenthesis around it... This makes me question if nose will actually work here as it clearly looks to be made for earlier versions of Python.
In the comments you mentioned that xx viewproject < answers.txt > output.txt works but you can't use it because answers depend on the output from the subprocess.
In general pexpect-like modules such as winpexpect (for Windows) could be used. Something like:
import re
import sys
from functools import partial
from winpexpect import EOF, winspawn as spawn
p = spawn('xx viewproject')
p.logfile = sys.stdout
patterns = ['the project:', re.escape('? [ynYN](n)'), EOF]
for found in iter(partial(p.expect, patterns), 2): # until EOF
if found == 0:
p.sendline(project_name)
elif found == 1:
filename = get_filename_from_prompt(p.before) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'no') # a dict
p.sendline(answer)
If the prompts are terminated with a newline (and the subprocess doesn't buffer them); you could read line by line using subprocess module directly:
from subprocess import Popen, PIPE
with Popen(["xx", "viewproject"], stdin=PIPE, stdout=PIPE,
universal_newlines=True) as p:
for line in p.stdout:
if line.startswith("Please enter the name of the project"):
answer = project_name
elif line.startswith("Would you like to recurse into the subproject"):
filename = get_filename_from_prompt(line) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'n') # a dict
else:
continue # skip it
print(answer, file=p.stdin) # provide answer
p.stdin.flush()
To test that you can read something from the xx using subprocess:
from subprocess import Popen, PIPE, STDOUT
with Popen(["xx", "viewproject"], bufsize=0,
stdin=PIPE, stdout=PIPE, stderr=STDOUT) as p:
print(repr(p.stdout.read(1)))
Yes, first of all you may create subprocess as an object by:
p = subprocess.Popen('xx viewproject', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, universal_newlines=True)
Then you'll have methods like communicate() available, for instance:
newline = os.linesep # [1]
commands = ['y', 'n', 'y', 'n', 'y']
p.communicate( newline.join( commands))
1 - os.linesep
Which will send all the answers at once (and hopefully it'll be enough) relying on the same order of question every time.
You may also try parsing p.stdout and then writing to p.stdin, but this may cause deadlock when one buffer will get full while waiting for another, so be careful with this. Luckily there are some complex examples on google.
Simple version would be:
p = Popen(...)
line = p.stdout.readline() # At this point, if child process will wait for stdin
# you have a deadlock on your hands
parse_line( line)
p.stdin.write( newline.join( commands).encode( 'utf-8'))
I would also consider rewriting:
p = subprocess.Popen('si viewproject --project=d:/Projects/test.pj', shell=True,
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
To:
p = subprocess.Popen( ['si', 'viewproject', '--project=d:/Projects/test.pj'],
shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
Unless you explicitly need Shell invocation.
I am trying to use pexpect module (version 3.3) with Python 3.4.0. I get an error
TypeError: must be str, not bytes
when I call child.expect method.
Actual code is standard example from pexpect documentation:
child = pexpect.spawn('ssh foo#bar.com')
index = child.expect([pexpect.TIMEOUT, pexpect.EOF, ssh_newkey, '.*password:'])
Exactly the same code works properly with pexpect module (version 3.1), and Python version 2.7.6.
Pexpect documentation on GitHub states that pexpect version 3.3 requires Python 2.6 or 3.2 or above. Does anybody know if pexpect does not work with Python 3 for some reason despite what is stated in documentation for this module?
This is traceback output that I get:
Traceback (most recent call last):
File "/home/sambo9/python/python3-pexpect.py", line 17, in <module>
main()
File "/home/sambo9/python/python3-pexpect.py", line 13, in main
child.expect('.*password:')
File "/usr/local/lib/python3.4/dist-packages/pexpect/__init__.py", line 1451, in expect
timeout, searchwindowsize)
File "/usr/local/lib/python3.4/dist-packages/pexpect/__init__.py", line 1466, in expect_list
timeout, searchwindowsize)
File "/usr/local/lib/python3.4/dist-packages/pexpect/__init__.py", line 1535, in expect_loop
c = self.read_nonblocking(self.maxread, timeout)
File "/usr/local/lib/python3.4/dist-packages/pexpect/__init__.py", line 985, in read_nonblocking
self._log(s, 'read')
File "/usr/local/lib/python3.4/dist-packages/pexpect/__init__.py", line 908, in _log
second_log.write(s)
File "/usr/lib/python3.4/idlelib/PyShell.py", line 1339, in write
raise TypeError('must be str, not ' + type(s).__name__)
TypeError: must be str, not bytes
Additionally, I also see "OpenSSH" box pop open via GUI on Ubuntu prompting for password when I run the script. This does not happen under Python 2.7.6. In Python 2.7, I can login into system without any manual interaction - everything happens automatically via script.
From the documentation:
# In Python 3, spawnu should be used to give str to stdout:
child = pexpect.spawnu('some_command')
child.logfile = sys.stdout
I'm running on Windows 7 x64. I followed the install documentation on Buildbot and did some research on the issue I'm having and haven't found a solution yet. When I do a force build, everything works fine. I'm using GitPoller. When it tries to poll for changes, an exception is thrown; why? Let me know if I can supply any more information. Here's what I'm getting on the master's twistd.log every 5 minutes:
2014-10-09 00:19:53-0700 [-] while polling for changes
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\util\misc.py", line 54, in start
d = self.method()
File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\changes\base.py", line 70, in doPoll
d = defer.maybeDeferred(self.poll)
File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 139, in maybeDeferred
result = f(*args, **kw)
File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 1237, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 1099, in _inlineCallbacks
result = g.send(result)
File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\changes\gitpoller.py", line 147, in poll
yield self._dovccmd('init', ['--bare', self.workdir])
File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\changes\gitpoller.py", line 292, in _dovccmd
[command] + args, path=path, env=os.environ)
File "C:\Python27\lib\site-packages\twisted\internet\utils.py", line 176, in getProcessOutputAndValue
reactor)
File "C:\Python27\lib\site-packages\twisted\internet\utils.py", line 30, in _callProtocolWithDeferred
reactor.spawnProcess(p, executable, (executable,)+tuple(args), env, path)
File "C:\Python27\lib\site-packages\twisted\internet\posixbase.py", line 358, in spawnProcess
return Process(self, processProtocol, executable, args, env, path)
File "C:\Python27\lib\site-packages\twisted\internet\_dumbwin32proc.py", line 195, in __init__
raise OSError(pwte)
exceptions.OSError: (2, 'CreateProcess', 'The system cannot find the file specified.')
Also, here's the relevant portion of my config file:
from buildbot.changes.gitpoller import GitPoller
c['change_source'] = []
c['change_source'].append(GitPoller(
repourl='https://github.com/solstice333/BuildbotTest.git',
branch='master',
pollinterval=300))
Any ideas?
I have similar issue with HgPoller. Try to specify full path to git
c['change_source'].append(GitPoller(
gitbin='full/path/to/git.exe',
repourl='https://github.com/solstice333/BuildbotTest.git',
branch='master',
pollinterval=300))
I think something wrong with twisted - this dont work with same error
PS Twisted use win32process.CreateProcess and MSDN says about it first argument: The string can specify the full path and file name of the module to execute or it can specify a partial name. In the case of a partial name, the function uses the current drive and current directory to complete the specification. The function will not use the search path.
from twisted.internet import utils
utils.getProcessOutputAndValue("hg.exe", ['init', "test_dir"])