Develop Reference

Python: Run command for another software in terminal

I am using a software that my lab has developed, lets call it cool_software. When I type cool_software on terminal, basically I get a new prompt cool_software > and I can imput commands to this software from the terminal.
Now I would like to automate this in Python, however I am not sure how to pass the cool_software commands onto it. Here's my MWE:
import os
os.system(`cool_software`)
os.system(`command_for_cool_software`)
The problem with the code above is that command_for_cool_software is executed in the usual unix shell, it is not executed by the cool_software.

Based on #Barmar suggestion from the comments, using pexpect is pretty neat. From the documentation:
The spawn class is the more powerful interface to the Pexpect system. You can use this to spawn a child program then interact with it by sending input and expecting responses (waiting for patterns in the child’s output).
This is a working example using the python prompt as an example:
import pexpect
child = pexpect.spawn("python") # mimcs running $python
child.sendline('print("hello")') # >>> print("hello")
child.expect("hello") # expects hello
print(child.after) # prints "hello"
child.close()
In your case, it will be like this:
import pexpect
child = pexpect.spawn("cool_software")
child.sendline(command_for_cool_software)
child.expect(expected_output) # catch the expected output
print(child.after)
child.close()
NOTE
child.expect() matches only what you expect. If you don't expect anything and want to get all the output since you started spawn, then you can use child.expect('.+') which would match everything.
This is what I got:
b'Python 3.8.10 (default, Jun 2 2021, 10:49:15) \r\n[GCC 9.4.0] on linux\r\nType "help", "copyright", "credits" or "license" for more information.\r\n>>> print("hello")\r\nhello\r\n>>> '

What's the Python equivalent of Julia's `#edit` macro?

In Julia, calling a function with the #edit macro from the REPL will open the editor and put the cursor at the line where the method is defined. So, doing this:
julia> #edit 1 + 1
jumps to julia/base/int.jl and puts the cursor on the line:
(+)(x::T, y::T) where {T<:BitInteger} = add_int(x, y)
As does the function form: edit(+, (Int, Int))
Is there an equivalent decorator/function in Python that does the same from the Python REPL?

Disclaimer: In the Python ecosystem, this is not the job of the core language/runtime but rather tools such as IDEs. For example, the ipython shell has the ?? special syntax to get improved help including source code.
Python 3.8.5 (default, Jul 21 2020, 10:42:08)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.18.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import random
In [2]: random.uniform??
Signature: random.uniform(a, b)
Source:
def uniform(self, a, b):
"Get a random number in the range [a, b) or [a, b] depending on rounding."
return a + (b-a) * self.random()
File: /usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/random.py
Type: method
The Python runtime itself allows viewing source code of objects via inspect.getsource. This uses a heuristic to search the source code as available; the objects themselves do not carry their source code.
Python 3.8.5 (default, Jul 21 2020, 10:42:08)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import inspect
>>> print(inspect.getsource(inspect.getsource))
def getsource(object):
"""Return the text of the source code for an object.
The argument may be a module, class, method, function, traceback, frame,
or code object. The source code is returned as a single string. An
OSError is raised if the source code cannot be retrieved."""
lines, lnum = getsourcelines(object)
return ''.join(lines)
It is not possible to resolve arbitrary expressions or statements to their source; since all names in Python are resolved dynamically, the vast majority of expressions does not have a well-defined implementation unless executed. A debugger, e.g. as provided by pdb.set_trace(), allows inspecting the expression as it is executed.

In most IDEs like PyCharm or VSCode you can Ctrl+ click on a function / class to get its definition, even if it is in the core language or a 3rd party library (in VSCode, this also works in Julia btw.).
A limitation is that this only works for "pure Python" code, C library code, etc. is not shown.

How can I call `IPython.start_ipython()` with my own banner?

What works
When calling IPython.embed(), one can pass banner1, banner2 or header to customize the message that appears before the interactive session, like this:
import IPython
IPython.embed(banner2="*** Welcome! ***")
With the result:
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
Type "copyright", "credits" or "license" for more information.
IPython 3.2.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
*** Welcome! ***
In [1]:
What doesn't work
When using IPython.start_ipython(), instead of IPython.embed() in the invocation above, I couldn't find any parameters that would influence the banner, except display_banner=False to omit the it entirely.
The best I could do was to mangle argv, to change the configuration, like:
import sys, IPython
argv = (
sys.argv[1:] +
['--TerminalInteractiveShell.banner2=*** Welcome! ***']
)
IPython.start_ipython(argv=argv)
This is usable but looks contrived.
I suppose I could also inherit from IPython.ipapp.TerminalInteractiveShell in my code and override .banner1 or .banner2, but this feels like overkill.
The Question
All I want is a way to pass banner2 into IPython.start_ipython().
Is there a more straightforward way?
More Technical details
The use case is to create a script that starts an IPython console session with some pre-defined variables for controlling an application with a fairly involved setup. And explain how to use the setup.
Something like:
import sys, myapp, IPython
explain_usage_of_session = """
You can use session.blah() to frobnicate the foobaringo
"""
session = myapp.MyComplicatedSessionFactory(including=
configuration.params(from_file))
sys.exit(
IPython.start_ipython(user_ns=dict(session=session),
banner2=explain_usage_of_session)
)
Constraints
The more specific use-case is that this script is being generated automatically by buildout's zc.recipe.egg, which locates IPython.start_ipython using IPython [console_scripts] entry point, so I'm limited in the amount of customization I can actually pass into the script, and I can't use IPython.embed() directly.
The super duper plus specific use-case is that I'm actually using anybox.recipe.odoo, which wraps zc.recipe.egg. The end result is that I'm even more limited in how the script is built.
Basically I can just set the parameters that are passed into IPython.start_ipython() call as with the arguments option of zc.recipe.egg, and nothing else. In particular, I can't use the initialization option of zc.recipe.egg.
And I'd rather not have to write my own entry-point.

As #Thomas K said, you can create an IPython.Config instance and set the banner2:
from IPython import start_ipython
from traitlets.config.loader import Config
c = Config()
c.TerminalInteractiveShell.banner2 = '*** Welcome! ***'
start_ipython(config=c)
The result:
$ python start_with_banner.py
Python 2.7.11+ (default, Mar 30 2016, 21:00:42)
Type "copyright", "credits" or "license" for more information.
IPython 2.4.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
*** Welcome! ***
In [1]:
Ftr: the Config constructor accepts kwargs:
c = Config(TerminalInteractiveShell={'banner2': '*** Welcome! ***'})
Hth,
dtk
Update: For versions before ipython 5.x, you could directly from IPython import Config.

How to execute multi-line statements within Python's own debugger (PDB)

So I am running a Python script within which I am calling Python's debugger, PDB by writing:
import ipdb; ipdb.set_trace()
(iPython's version of PDB, though for the matter I don't think it makes a difference; I use it for the colored output only).
Now, when I get to the debugger I want to execute a multi-line statement such as an if clause or a for loop but as soon as I type
if condition:
and hit the return key, I get the error message *** SyntaxError: invalid syntax (<stdin>, line 1)
How can one execute multi-line statements within PDB? If not possible is there a way around this to still executing an if clause or a for loop?

You could do this while in pdb to launch a temporary interactive Python session with all the local variables available:
(pdb) !import code; code.interact(local=vars())
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
When you're done, use Ctrl-D to return to the regular pdb prompt.
Just don't hit Ctrl-C, that will terminate the entire pdb session.

In python3 ipdb (and pdb) have a command called interact. It can be used to:
Start an interactive interpreter (using the code module) whose global namespace contains all the (global and local) names found in the current scope.
To use it, simply enter interact at the pdb prompt. Among other things, it's useful for applying code spanning multiple lines, and also for avoiding accidental triggering of other pdb commands.

My recommendation is to use IPython embedding.
ipdb> from IPython import embed; embed()

Inside the Python (2.7.1) interpreter or debugger (import pdb), you can execute a multi-line statement with the following syntax.
for i in range(5): print("Hello"); print("World"); print(i)
Note: When I'm inside the interpreter, I have to hit return twice before the code will execute. Inside the debugger, however, I only have to hit return once.

There is the special case if you want a couple of commands be executed when hitting a break point. Then there is the debugger command commands. It allows you to enter multiple lines of commands and then end the whole sequence with the end key word. More with (pdb) help commands.

I don't know if you can do this, that'd be a great feature for ipdb though. You can use list comprehensions of course, and execute simple multi-line expressions like:
if y == 3: print y; print y; print y;
You could also write some functions beforehand to do whatever it is you need done that would normally take multiple lines.

python: find out if running in shell or not (e.g. sun grid engine queue)

is there a way to find out from within a python program if it was started in a terminal or e.g. in a batch engine like sun grid engine?
the idea is to decide on printing some progress bars and other ascii-interactive stuff, or not.
thanks!
p.

The standard way is isatty().
import sys
if sys.stdout.isatty():
print("Interactive")
else:
print("Non-interactive")

You can use os.getppid() to find out the process id for the parent-process of this one, and then use that process id to determine which program that process is running. More usefully, you could use sys.stdout.isatty() -- that doesn't answer your title question but appears to better solve the actual problem you explain (if you're running under a shell but your output is piped to some other process or redirected to a file you probably don't want to emit "interactive stuff" on it either).

Slightly shorter:
import sys
sys.stdout.isatty()

I have found the following to work on both Linux and Windows, in both the normal Python interpreter and IPython (though I can't say about IronPython):
isInteractive = hasattr(sys, 'ps1') or hasattr(sys, 'ipcompleter')
However, note that when using ipython, if the file is specified as a command-line argument it will run before the interpreter becomes interactive. See what I mean below:
C:\>cat C:\demo.py
import sys, os
# ps1=python shell; ipcompleter=ipython shell
isInteractive = hasattr(sys, 'ps1') or hasattr(sys, 'ipcompleter')
print isInteractive and "This is interactive" or "Automated"
C:\>python c:\demo.py
Automated
C:\>python
>>> execfile('C:/demo.py')
This is interactive
C:\>ipython C:\demo.py
Automated # NOTE! Then ipython continues to start up...
IPython 0.9.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
In [2]: run C:/demo.py
This is interactive # NOTE!
HTH