I am using a software that my lab has developed, lets call it cool_software. When I type cool_software on terminal, basically I get a new prompt cool_software > and I can imput commands to this software from the terminal.
Now I would like to automate this in Python, however I am not sure how to pass the cool_software commands onto it. Here's my MWE:
import os
os.system(`cool_software`)
os.system(`command_for_cool_software`)
The problem with the code above is that command_for_cool_software is executed in the usual unix shell, it is not executed by the cool_software.
Based on #Barmar suggestion from the comments, using pexpect is pretty neat. From the documentation:
The spawn class is the more powerful interface to the Pexpect system. You can use this to spawn a child program then interact with it by sending input and expecting responses (waiting for patterns in the child’s output).
This is a working example using the python prompt as an example:
import pexpect
child = pexpect.spawn("python") # mimcs running $python
child.sendline('print("hello")') # >>> print("hello")
child.expect("hello") # expects hello
print(child.after) # prints "hello"
child.close()
In your case, it will be like this:
import pexpect
child = pexpect.spawn("cool_software")
child.sendline(command_for_cool_software)
child.expect(expected_output) # catch the expected output
print(child.after)
child.close()
NOTE
child.expect() matches only what you expect. If you don't expect anything and want to get all the output since you started spawn, then you can use child.expect('.+') which would match everything.
This is what I got:
b'Python 3.8.10 (default, Jun 2 2021, 10:49:15) \r\n[GCC 9.4.0] on linux\r\nType "help", "copyright", "credits" or "license" for more information.\r\n>>> print("hello")\r\nhello\r\n>>> '
Related
Normally, in Python shells I can press Tab twice to get a list of prompts.
On the other hand, in gdb's Python shell (pi or python-interactive command), there's only gdb-style completion.
Example session:
$ gdb -q
(gdb) pi
>>> gdb
<module 'gdb' from '/usr/share/gdb/python/gdb/__init__.py'>
>>> gdb.TabTab
... nothing ...
>>> show TabTab
Display all 148 possibilities? (y or n)
ada exec-direction record
agent exec-done-display remote
annotate exec-file-mismatch remoteaddresssize
[...]
Python auto complete should be at least like this.
$ python
Python 3.X.Y ...
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.TabTab
sys.abiflags sys.hash_info
sys.addaudithook( sys.hexversion
sys.api_version sys.implementation
[...]
How can I get the same/similar thing in gdb? in particular IPython shell with tab completion is fine.
Failed attempts:
This solution
import readline
import rlcompleter
readline.parse_and_bind("tab: complete")
makes the shell output a literal tab when Tab is pressed after sys. or similar.
At least it does work for identifier tab completion (aTabTab) does list some entries)
Looks like this is because of some interaction with gdb -- get_completer_delims get reset to some value every time, and if the code above is run then the tab completion outside gdb switches to "Python mode" as well.
Use background_zmq_ipython causes segmentation fault, because some gdb API (such as gdb.Value) cannot be read from outside the main thread.
Use IPython.embed() also make Tab output a literal tab character.
The official gdb documentation https://sourceware.org/gdb/current/onlinedocs/gdb/Completion.html doesn't mention anything about Python.
There are a few ways I've figured out.
I can't figure out any way to use the built-in readline library.
See the failed attempts in the question for more details.
Reset stdout and stderr before calling IPython.embed.
import sys
sys.stdout=sys.__stdout__
sys.stderr=sys.__stderr__
import IPython
IPython.embed(colors="neutral")
Remember to reset stdout and stderr afterwards to avoid possible issues.
Reference:
How to recognize whether a script is running on a tty?
IPython.embed() does not use terminal colors
IPython will only use tab-completion and color when all of stdin, stdout and stderr are tty devices.
By default gdb sys.stdout and sys.stderr are gdb wrappers (so that gdb can do "press enter to continue"
when pagination limit is exceeded)
Start a kernel, and start a console separately.
import IPython
IPython.embed_kernel()
Read the console output on how to connect, and how to exit the terminal from the remote console.
Using my other answer it's also possible to exit the terminal remotely programmatically.
Start a kernel (the complex way)
Read the source code of IPython to figure out how to start a kernel manually, and get the connection file path in the process.
import threading
import subprocess
import IPython
from ipykernel.kernelapp import IPKernelApp
import sys
app = IPKernelApp.instance()
app.initialize([])
app.kernel.user_module = sys.modules[__name__]
app.kernel.user_ns = locals()
app.shell.set_completer_frame()
def console_thread_run():
subprocess.run(["jupyter-console", "--no-confirm-exit", "--existing",
app.abs_connection_file
])
app.kernel.do_shutdown(restart=False)
console_thread=threading.Thread(target=console_thread_run)
console_thread.start()
app.start()
Start a kernel using background_zmq_ipython (accesses internal property, might break at any time).
The main difference is that sys.stdin, sys.stdout etc. are not affected. See background_zmq_ipython
documentation and ipython - Provide remote shell for Python script - Stack Overflow for more details.
import subprocess
import logging
import threading
from background_zmq_ipython import IPythonBackgroundKernelWrapper
kernel_wrapper = IPythonBackgroundKernelWrapper(
banner="", # default value is "Hello from background-zmq-ipython."
user_ns=globals(),
logger=logging.Logger("IPython", level=logging.INFO)
# no handler
# otherwise it will print "To connect another client to this IPython kernel" ...
)
kernel_wrapper.thread=threading.main_thread() # workaround for assertions
subprocess.Popen(["python", "-c",
(
"from jupyter_console.app import ZMQTerminalIPythonApp;"
"app = ZMQTerminalIPythonApp();"
"app.initialize();"
"app.shell.own_kernel=True;"
"app.start();"
),
"--no-confirm-exit",
"--existing", kernel_wrapper.connection_filename
])
kernel_wrapper._thread_loop()
Also show how to change the message "keeping kernel alive" to "Shutting down kernel".
My question is more theoretical than practical, I've found more answers that explains how but not why should we use a list in a subprocess.Popen call.
For example as is known:
Python 2.7.10 (default, Oct 14 2015, 16:09:02)
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> cmd = subprocess.Popen(["python", "-V"], stdout=subprocess.PIPE)
Python 2.7.10
Then I was messing around in UNIX and found something interesting:
mvarge#ubuntu:~$ strace -f python -V 2>&1
execve("/usr/bin/python", ["python", "-V"], [/* 29 vars */]) = 0
Probably both execve and the list model that subprocess use are someway related, but can anyone give a good explanation for this?
Thanks in advance.
The underlying C-level representation is a *char [] array. Representing this as a list in Python is just a very natural and transparent mapping.
You can use a string instead of a list with shell=True; the shell is then responsible for parsing the command line into a * char [] array. However, the shell adds a number of pesky complexities; see the many questions for why you want to avoid shell=True for a detailed explanation.
The command line arguments argv and the environment envp are just two of many OS-level structures which are essentially a null-terminated arrays of strings.
A process is an OS level abstraction — to create a process, you have to use OS API that dictates what you should use. It is not necessary to use a list e.g., a string (lpCommandLine) is the native interface on Windows (CreateProcess()). POSIX uses execv() and therefore the native interface is a sequence of arguments (argv). Naturally, subprocess Python module uses these interfaces to run external commands (create new processes).
The technical (uninsteresting) answer is that in "why we must", the "must" part is not correct as Windows demonstrates.
To understand "why it is", you could ask the creators of CreateProcess(), execv() functions.
To understand "why we should" use a list, look at the table of contents for Unix (list) and Windows (string): How Command Line Parameters Are Parsed — the task that should be simple is complicated on Windows.
The main difference is that on POSIX the caller is responsible for splitting a command line into separate parameters. While on Windows the command itself parses its parameters. Different programs may and do use different algorithms to parse the parameters. subprocess module uses MS C runtime rules (subprocess.list2cmdline()), to combine args list into the command line. It is much harder for a programmer to understand how the parameters might be parsed on Windows.
Let's say my shell script returns a value of '19' when it is run. I'd like to store that value (without any return value of 0 or empty lines) into a variable in my python code for use later.
There are many questions here with similar reference to mine but I have yet to find a solution where the shell script is returning me '19' without a extra return value of 0 or new lines.
Using subprocess.call('bash TestingCode', shell=True) in the python code returns me exactly what I want however when I store this command in a variable and then print the variable, it prints with an extra 0.
answer = subprocess.call('bash TestingCode', shell=True)
print answer
>>19
>>0
I then tried an example from this question: How to return a value from a shell script in a python script
However it returns me an extra empty line instead.
answer = subprocess.check_output('bash TestingCode', shell=True)
print answer
>>19
>>
I really appreciate the help!
UPDATE: TestingCode script
#!/bin/bash
num=19
echo $num
Just call it like this:
import subprocess
answer = subprocess.check_output('bash TestingCode', shell=True)
answer = answer.rstrip()
The reason is that your shell script is printing 19 followed by a new line. The return value from subprocess.check_output() will therefore include the new line produced by the shell script. Calling str.rstrip() will remove any trailing whitespace, leaving just '19' in this case.
Try calling a subprocess.Popen, it returns without the 0.
This works for me. I suspect you have a problem in your shell script that is causing the output.
$ cat test.sh
#!/bin/sh
exit 19
(0) austin#Austins-Mac-8:~
$ python2.7
Python 2.7.10 (default, Aug 22 2015, 20:33:39)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.call('bash test.sh', shell=True)
19
>>>
What works
When calling IPython.embed(), one can pass banner1, banner2 or header to customize the message that appears before the interactive session, like this:
import IPython
IPython.embed(banner2="*** Welcome! ***")
With the result:
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
Type "copyright", "credits" or "license" for more information.
IPython 3.2.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
*** Welcome! ***
In [1]:
What doesn't work
When using IPython.start_ipython(), instead of IPython.embed() in the invocation above, I couldn't find any parameters that would influence the banner, except display_banner=False to omit the it entirely.
The best I could do was to mangle argv, to change the configuration, like:
import sys, IPython
argv = (
sys.argv[1:] +
['--TerminalInteractiveShell.banner2=*** Welcome! ***']
)
IPython.start_ipython(argv=argv)
This is usable but looks contrived.
I suppose I could also inherit from IPython.ipapp.TerminalInteractiveShell in my code and override .banner1 or .banner2, but this feels like overkill.
The Question
All I want is a way to pass banner2 into IPython.start_ipython().
Is there a more straightforward way?
More Technical details
The use case is to create a script that starts an IPython console session with some pre-defined variables for controlling an application with a fairly involved setup. And explain how to use the setup.
Something like:
import sys, myapp, IPython
explain_usage_of_session = """
You can use session.blah() to frobnicate the foobaringo
"""
session = myapp.MyComplicatedSessionFactory(including=
configuration.params(from_file))
sys.exit(
IPython.start_ipython(user_ns=dict(session=session),
banner2=explain_usage_of_session)
)
Constraints
The more specific use-case is that this script is being generated automatically by buildout's zc.recipe.egg, which locates IPython.start_ipython using IPython [console_scripts] entry point, so I'm limited in the amount of customization I can actually pass into the script, and I can't use IPython.embed() directly.
The super duper plus specific use-case is that I'm actually using anybox.recipe.odoo, which wraps zc.recipe.egg. The end result is that I'm even more limited in how the script is built.
Basically I can just set the parameters that are passed into IPython.start_ipython() call as with the arguments option of zc.recipe.egg, and nothing else. In particular, I can't use the initialization option of zc.recipe.egg.
And I'd rather not have to write my own entry-point.
As #Thomas K said, you can create an IPython.Config instance and set the banner2:
from IPython import start_ipython
from traitlets.config.loader import Config
c = Config()
c.TerminalInteractiveShell.banner2 = '*** Welcome! ***'
start_ipython(config=c)
The result:
$ python start_with_banner.py
Python 2.7.11+ (default, Mar 30 2016, 21:00:42)
Type "copyright", "credits" or "license" for more information.
IPython 2.4.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
*** Welcome! ***
In [1]:
Ftr: the Config constructor accepts kwargs:
c = Config(TerminalInteractiveShell={'banner2': '*** Welcome! ***'})
Hth,
dtk
Update: For versions before ipython 5.x, you could directly from IPython import Config.
So I am running a Python script within which I am calling Python's debugger, PDB by writing:
import ipdb; ipdb.set_trace()
(iPython's version of PDB, though for the matter I don't think it makes a difference; I use it for the colored output only).
Now, when I get to the debugger I want to execute a multi-line statement such as an if clause or a for loop but as soon as I type
if condition:
and hit the return key, I get the error message *** SyntaxError: invalid syntax (<stdin>, line 1)
How can one execute multi-line statements within PDB? If not possible is there a way around this to still executing an if clause or a for loop?
You could do this while in pdb to launch a temporary interactive Python session with all the local variables available:
(pdb) !import code; code.interact(local=vars())
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
When you're done, use Ctrl-D to return to the regular pdb prompt.
Just don't hit Ctrl-C, that will terminate the entire pdb session.
In python3 ipdb (and pdb) have a command called interact. It can be used to:
Start an interactive interpreter (using the code module) whose global namespace contains all the (global and local) names found in the current scope.
To use it, simply enter interact at the pdb prompt. Among other things, it's useful for applying code spanning multiple lines, and also for avoiding accidental triggering of other pdb commands.
My recommendation is to use IPython embedding.
ipdb> from IPython import embed; embed()
Inside the Python (2.7.1) interpreter or debugger (import pdb), you can execute a multi-line statement with the following syntax.
for i in range(5): print("Hello"); print("World"); print(i)
Note: When I'm inside the interpreter, I have to hit return twice before the code will execute. Inside the debugger, however, I only have to hit return once.
There is the special case if you want a couple of commands be executed when hitting a break point. Then there is the debugger command commands. It allows you to enter multiple lines of commands and then end the whole sequence with the end key word. More with (pdb) help commands.
I don't know if you can do this, that'd be a great feature for ipdb though. You can use list comprehensions of course, and execute simple multi-line expressions like:
if y == 3: print y; print y; print y;
You could also write some functions beforehand to do whatever it is you need done that would normally take multiple lines.