I am using python3 and subprocess.Popen to spawn a process of bash and invoking the Python3 interpreter again through the standard interpreter.
bash -i states:
-s If the -s option is present, or if no arguments remain after
option processing, then commands are read from the standard
input. This option allows the positional parameters to be
set when invoking an interactive shell.
This is a minimized example but it mainly bakes down to the following code:
import subprocess
import sys
p = subprocess.Popen(["bash", "-s"], stdin=subprocess.PIPE,stderr=sys.stderr, stdout=sys.stdout)
p.stdin.write(b"python3\n")
p.stdin.flush()
print("Done")
The output is simply "Done". Any suggestions how I need to handle the stdin pipes in order to let the interactive shell pop up inside the newly executed python3 interpreter?
Actual output
% python3 test.py
Done
Expected output:
% python3 test.py
Python 3.10.8 (main, Oct 13 2022, 10:17:43) [Clang 14.0.0 (clang-1400.0.29.102)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Related
I'm on a Ubuntu 20.04 system, and I'm using Python 3.8 to write a script that does multiple things using configurable lines of bash, but one of them is that it creates desktop shortcuts.
This single-line command creates a desktop shortcut, and works flawlessly when I execute it directly in my terminal:
echo "[Desktop Entry]"$'\n'"Type=Application"$'\n'"Name[en_US]=Documents"$'\n'"Exec=pcmanfm ~/Documents"$'\n'"Icon=system-file-manager" > ~/Desktop/1;
However, when I execute it in Python, like so:
foobar.py
rl = """echo "[Desktop Entry]"$'\n'"Type=Application"$'\n'"Name[en_US]=Documents"$'\n'"Exec=pcmanfm ~/Documents"$'\n'"Icon=system-file-manager" > ~/Desktop/1;"""
subprocess.run(rl, shell=True)
...instead of creating a desktop shortcut with the proper name, icon, and action, it creates an empty file that contains the following text:
0.txt:
[Desktop Entry]$\nType=Application$\nName[en_US]=Sign Out$\nExec=/home/agent/QaSC/login/login.bin$\nIcon=system-switch-user
Is there any particular reason why Python would be handling the newline characters differently than the bash shell does, and if so, how can I resolve this problem?
$'...' is a bash extension, but the default shell used when shell=True is specified is sh. Use the executable option to specify an alternate shell.
subprocess.run(rl, shell=True, executable='/bin/bash')
Since the argument to echo has quotes, it could contain literal newlines at the command line, and therefore also in the Python process. I see no reason to use the Bash extension $'\n' syntax.
$ echo "foo
> bar"
foo
bar
$ python
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.run('echo "foo\nbar"', shell=True)
foo
bar
CompletedProcess(args='echo "foo\nbar"', returncode=0)
>>>
(It's also unclear why the Python code doesn't just write a file in the normal way, but I assume the command in the OP is essentially a placeholder.)
When I run proc = subprocess.Popen('docker', stdout=subprocess.PIPE) on OSX 10.14.6 (18G84) with Python 3.7.4(installed via Homebrew) and docker 18.09.2(build 6247962), the stdout is printed in the console instead of being redirected.(i.e. proc.stdout.readlines() == [])
This only happens to 'docker' - another subprocess, for example 'ls', returned something via proc.stdout.readlines() instead of printing its output on the console.
Also, I made sure this is OSX-specific problem by running the same command on the EC2 instance(Amazon AMI 2, Docker 18.06.1-ce, Python 2.7.16&3.7.3), and proc.stdout.readlines() of docker subprocess returned the expeced result(no output in console, proc.stdout.readlines() contains docker output).
I googled and found a SO post Problems capturing Python subprocess output on Mac OS X but it seems like it's python problem(typo in python), not OSX problem.
Why is proc.stdout.readlines() returning nothing for 'docker'?
More details:
I was using thefuck open source cli tool, and one of its command was not working (specifically fixing docker typo commands.)
So I decided to fix it myself, and debugging it led me to this code:
...
import subprocess
...
def get_docker_commands():
proc = subprocess.Popen('docker', stdout=subprocess.PIPE)
lines = [line.decode('utf-8') for line in proc.stdout.readlines()]
These lines, where docker is run as subprocess and then its stdout is read by proc.stdout.readlines() was the source of the problem.
So I tried similar command in the python interactive console and it was like this:
(on Mac OS)
Python 3.7.4 (default, Jul 9 2019, 18:13:23)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> proc = subprocess.Popen('docker', stdout=subprocess.PIPE)
(docker help output in console)
>>> proc.stdout.readlines()
[]
>>> proc = subprocess.Popen('ls', stdout=subprocess.PIPE)
>>> proc.stdout.readlines()
(ls of directory '/' as python list)
(On EC2 Instance)
Python 3.7.3 (default, Jun 24 2019, 19:20:54)
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> proc = subprocess.Popen('docker', stdout=subprocess.PIPE)
>>> proc.stdout.readlines()
(docker output as python list)
I did not include the result but the python2's results are the same. (both Mac&EC2 are Python 2.7.16)
While fooling around with the OS module in the python interpreter (run inside the shell on a Linux system), I noticed it is possible to do the following:
>>> os.system("python") #execute run python command in enclosing shell
Producing the following output, indicating a new python REPL session:
Python 2.7.9 (default, Apr 2 2015, 15:34:55)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> #*blinking cursor*
From here, its possible to once again make a system call to start a new session of Python, from which I can make the system call again, etc. These Python environments seem independent from each other in that variables aren't shared amongst sessions and that the system calls are treated equivalently.
These sessions seem to run inside each other, at least to some extent, rather than in parallel, as evidenced by the outcome of the quit() function:
>>> os.system("python")
Python 2.7.9 (default, Apr 2 2015, 15:34:55)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
0
>>> quit()
0
>>> quit()
0
>>> quit()
shaked#shaked-ThinkPad-X220:~/Desktop$ python
Yet, a quick examination (>>> os.system("ps -e")) of the ongoing processes from the shell reveals a new sh for each python interpreter running:
11802 ? 00:00:00 kworker/u16:0
11803 pts/3 00:00:00 sh
11804 pts/3 00:00:00 python
11806 pts/3 00:00:00 sh
11807 pts/3 00:00:00 python
11810 pts/3 00:00:00 sh
11811 pts/3 00:00:00 python
11813 pts/3 00:00:00 sh
11814 pts/3 00:00:00 ps
Can anyone explain this (seemingly) strange behaviour in terms of underlying system processes? That is, are these sessions running in parallel or from within each other?
Apologies if this question has come up before, but I wasn't sure how others might have presented it.
When you use os.system, it performs the command in a subshell, something like /bin/sh -c 'yourcommand'*. Thus, it's completely sensible that you would get the behavior you describe. It's not different from running:
/bin/sh -c 'python'
in any shell.
*with mild differences on Windows, please read the documentation.
As per the documentation of os.system():
Execute the command (a string) in a subshell.
os.system() forks, executes a subshell passing the argument to this subshell and waits for the subshell to exit.
The subshell executes the command and waits for it to terminate, hence the process tree is:
- python
\-- sh
\-- python
\-- sh
\-- python
Python's os.system starts a new system process. When you pass "python" to it, it starts a new Python interpreter, that is independent from the one os.system was called from.
I can get a value n when I run a shell command using os.system in the python script, but I also need to sum it up to get a total number for subsequent computation in the python script,
total=0
for i in xrange(1,8):
os.system('n=$(qstat -n1 | grep -o node'+str(i)+' | wc -l) && echo $n')
Is it possible? Also is it possible to use python variable in shell command, something like
os.system('echo $total')
Use the shell's exportcommand:
$ export ABC=1 # Set and export var in shell
$ bash # Start another shell
$ echo $ABC # variable is still here
1
$ python # Start python still in the deeper shell
Python 2.7.2 (default, Oct 11 2012, 20:14:37)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from os import environ # import environnement
>>> environ['ABC'] # And here's the variable again (it's a string because the shell doesn't have types)
'1'
You can subprocess module's check_output method like this
import subprocess
print sum(int(subprocess.check_output(["/bin/sh", "-c", "n=`expr {} + 1` && echo $n".format(i)])) for i in range(10))
Output
55
I'm trying to, from a command line, open an instance of konsole and run a python script. I'm trying:
konsole -hold -e 'python -i hello.py'
The behaviour I'm getting is that a persistent konsole opens, and I am dropped into python, but the script does not run.
Python 2.7.2+ (default, Oct 4 2011, 20:03:08)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
What do I need to do to get the python script to run in the konsole window?
jsbueno's solution is the correct one. However, as described here, you can also do something like this ...
konsole --hold -e /bin/sh -c "python -i hello.py"
P.S. you'll need to specify --workdir (before the -e arg), or provide the full path to the python script, if it's not always in the initial working dir of konsole. But, you probably already knew that.
The problem is the way "konsole" uses the parameters after the -e switch - it seems like it simply pass them in a call that does not interpret the space separators as parameter separators.
However, if you don't put your call parameters inside quotes it will work - that is, simply:
konsole --hold -e python -i hello.py
(I just tested it here)