Develop Reference

Python: Run command for another software in terminal

I am using a software that my lab has developed, lets call it cool_software. When I type cool_software on terminal, basically I get a new prompt cool_software > and I can imput commands to this software from the terminal.
Now I would like to automate this in Python, however I am not sure how to pass the cool_software commands onto it. Here's my MWE:
import os
os.system(`cool_software`)
os.system(`command_for_cool_software`)
The problem with the code above is that command_for_cool_software is executed in the usual unix shell, it is not executed by the cool_software.

Based on #Barmar suggestion from the comments, using pexpect is pretty neat. From the documentation:
The spawn class is the more powerful interface to the Pexpect system. You can use this to spawn a child program then interact with it by sending input and expecting responses (waiting for patterns in the child’s output).
This is a working example using the python prompt as an example:
import pexpect
child = pexpect.spawn("python") # mimcs running $python
child.sendline('print("hello")') # >>> print("hello")
child.expect("hello") # expects hello
print(child.after) # prints "hello"
child.close()
In your case, it will be like this:
import pexpect
child = pexpect.spawn("cool_software")
child.sendline(command_for_cool_software)
child.expect(expected_output) # catch the expected output
print(child.after)
child.close()
NOTE
child.expect() matches only what you expect. If you don't expect anything and want to get all the output since you started spawn, then you can use child.expect('.+') which would match everything.
This is what I got:
b'Python 3.8.10 (default, Jun 2 2021, 10:49:15) \r\n[GCC 9.4.0] on linux\r\nType "help", "copyright", "credits" or "license" for more information.\r\n>>> print("hello")\r\nhello\r\n>>> '

What's the Python equivalent of Julia's `#edit` macro?

In Julia, calling a function with the #edit macro from the REPL will open the editor and put the cursor at the line where the method is defined. So, doing this:
julia> #edit 1 + 1
jumps to julia/base/int.jl and puts the cursor on the line:
(+)(x::T, y::T) where {T<:BitInteger} = add_int(x, y)
As does the function form: edit(+, (Int, Int))
Is there an equivalent decorator/function in Python that does the same from the Python REPL?

Disclaimer: In the Python ecosystem, this is not the job of the core language/runtime but rather tools such as IDEs. For example, the ipython shell has the ?? special syntax to get improved help including source code.
Python 3.8.5 (default, Jul 21 2020, 10:42:08)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.18.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import random
In [2]: random.uniform??
Signature: random.uniform(a, b)
Source:
def uniform(self, a, b):
"Get a random number in the range [a, b) or [a, b] depending on rounding."
return a + (b-a) * self.random()
File: /usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/random.py
Type: method
The Python runtime itself allows viewing source code of objects via inspect.getsource. This uses a heuristic to search the source code as available; the objects themselves do not carry their source code.
Python 3.8.5 (default, Jul 21 2020, 10:42:08)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import inspect
>>> print(inspect.getsource(inspect.getsource))
def getsource(object):
"""Return the text of the source code for an object.
The argument may be a module, class, method, function, traceback, frame,
or code object. The source code is returned as a single string. An
OSError is raised if the source code cannot be retrieved."""
lines, lnum = getsourcelines(object)
return ''.join(lines)
It is not possible to resolve arbitrary expressions or statements to their source; since all names in Python are resolved dynamically, the vast majority of expressions does not have a well-defined implementation unless executed. A debugger, e.g. as provided by pdb.set_trace(), allows inspecting the expression as it is executed.

In most IDEs like PyCharm or VSCode you can Ctrl+ click on a function / class to get its definition, even if it is in the core language or a 3rd party library (in VSCode, this also works in Julia btw.).
A limitation is that this only works for "pure Python" code, C library code, etc. is not shown.

Import lPython to debug a program

Someone explain to me instead of verifying your python code program with some print function, it is suggested to use :
import lPython
lPython.embed()
As I don't have I lot of experience with python, could anyone be able to show me how it work with an example? And maybe how to understand each statement?

I believe you have a typo with IPython.
Here is an example:
a = 1
from IPython import embed # please notice the 'I' not the 'l'
embed()
c = 2
What this line of code does - it stops execution of your program and runs python session which allows you to introspect values. For instance if I run this code:
python tmp.py
It will stop the execution, run python and it will show you an ipython interpepator.
Python 2.7.13 (default, Dec 18 2016, 07:03:39)
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]:
If you type 'who' there you will list of all variables defined before ipython was called. See more information here: ipythondocs
however
I would strongly recommend to get familiar with pdb or ipdb.
Check documentation here pdbdocs. It is much more easier and gives you an ability of actual debugging of the code as in previous solution any changes you make - will not affect execution runtime.

Why we must use a list in subprocess.Popen?

My question is more theoretical than practical, I've found more answers that explains how but not why should we use a list in a subprocess.Popen call.
For example as is known:
Python 2.7.10 (default, Oct 14 2015, 16:09:02)
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> cmd = subprocess.Popen(["python", "-V"], stdout=subprocess.PIPE)
Python 2.7.10
Then I was messing around in UNIX and found something interesting:
mvarge#ubuntu:~$ strace -f python -V 2>&1
execve("/usr/bin/python", ["python", "-V"], [/* 29 vars */]) = 0
Probably both execve and the list model that subprocess use are someway related, but can anyone give a good explanation for this?
Thanks in advance.

The underlying C-level representation is a *char [] array. Representing this as a list in Python is just a very natural and transparent mapping.
You can use a string instead of a list with shell=True; the shell is then responsible for parsing the command line into a * char [] array. However, the shell adds a number of pesky complexities; see the many questions for why you want to avoid shell=True for a detailed explanation.
The command line arguments argv and the environment envp are just two of many OS-level structures which are essentially a null-terminated arrays of strings.

A process is an OS level abstraction — to create a process, you have to use OS API that dictates what you should use. It is not necessary to use a list e.g., a string (lpCommandLine) is the native interface on Windows (CreateProcess()). POSIX uses execv() and therefore the native interface is a sequence of arguments (argv). Naturally, subprocess Python module uses these interfaces to run external commands (create new processes).
The technical (uninsteresting) answer is that in "why we must", the "must" part is not correct as Windows demonstrates.
To understand "why it is", you could ask the creators of CreateProcess(), execv() functions.
To understand "why we should" use a list, look at the table of contents for Unix (list) and Windows (string): How Command Line Parameters Are Parsed — the task that should be simple is complicated on Windows.
The main difference is that on POSIX the caller is responsible for splitting a command line into separate parameters. While on Windows the command itself parses its parameters. Different programs may and do use different algorithms to parse the parameters. subprocess module uses MS C runtime rules (subprocess.list2cmdline()), to combine args list into the command line. It is much harder for a programmer to understand how the parameters might be parsed on Windows.

python: find out if running in shell or not (e.g. sun grid engine queue)

is there a way to find out from within a python program if it was started in a terminal or e.g. in a batch engine like sun grid engine?
the idea is to decide on printing some progress bars and other ascii-interactive stuff, or not.
thanks!
p.

The standard way is isatty().
import sys
if sys.stdout.isatty():
print("Interactive")
else:
print("Non-interactive")

You can use os.getppid() to find out the process id for the parent-process of this one, and then use that process id to determine which program that process is running. More usefully, you could use sys.stdout.isatty() -- that doesn't answer your title question but appears to better solve the actual problem you explain (if you're running under a shell but your output is piped to some other process or redirected to a file you probably don't want to emit "interactive stuff" on it either).

Slightly shorter:
import sys
sys.stdout.isatty()

I have found the following to work on both Linux and Windows, in both the normal Python interpreter and IPython (though I can't say about IronPython):
isInteractive = hasattr(sys, 'ps1') or hasattr(sys, 'ipcompleter')
However, note that when using ipython, if the file is specified as a command-line argument it will run before the interpreter becomes interactive. See what I mean below:
C:\>cat C:\demo.py
import sys, os
# ps1=python shell; ipcompleter=ipython shell
isInteractive = hasattr(sys, 'ps1') or hasattr(sys, 'ipcompleter')
print isInteractive and "This is interactive" or "Automated"
C:\>python c:\demo.py
Automated
C:\>python
>>> execfile('C:/demo.py')
This is interactive
C:\>ipython C:\demo.py
Automated # NOTE! Then ipython continues to start up...
IPython 0.9.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
In [2]: run C:/demo.py
This is interactive # NOTE!
HTH