I have the following function that is used to execute system commands in Python:
def engage_command(
command = None
):
#os.system(command)
return os.popen(command).read()
I am using the os module instead of the subprocess module because I am dealing with a single environment in which I am interacting with many environment variables etc.
How can I use Bash with this type of function instead of the default sh shell?
output = subprocess.check_output(command, shell=True, executable='/bin/bash')
os.popen() is implemented in terms of subprocess module.
I am dealing with a single environment in which I am interacting with many environment variables etc.
each os.popen(cmd) call creates a new /bin/sh process, to run cmd shell command.
Perhaps, it is not obvious from the os.popen() documentation that says:
Open a pipe to or from command cmd
"open a pipe" does not communicate clearly: "start a new shell process with a redirected standard input or output" -- your could report a documentation issue.
If there is any doubt; the source confirms that each successful os.popen() call creates a new child process
the child can't modify its parent process environment (normally).
Consider:
import os
#XXX BROKEN: it won't work as you expect
print(os.popen("export VAR=value; echo ==$VAR==").read())
print(os.popen("echo ==$VAR==").read())
Output:
==value==
====
==== means that $VAR is empty in the second command because the second command runs in a different /bin/sh process from the first one.
To run several bash commands inside a single process, put them in a script or pass as a string:
output = check_output("\n".join(commands), shell=True, executable='/bin/bash')
Example:
#!/usr/bin/env python
from subprocess import check_output
output = check_output("""
export VAR=value; echo ==$VAR==
echo ==$VAR==
""", shell=True, executable='/bin/bash')
print(output.decode())
Output:
==value==
==value==
Note: $VAR is not empty here.
If you need to generate new commands dynamically (based on the output from the previous commands); it creates several issues and some of the issues could be fixed using pexpect module: code example.
Related
I would like to run this multiline shell commands:
echo 'a=?'
read a
echo "a=$a"
from a python script, using the subprocess.call() method.
I wrote this, in test.py file:
import shlex, subprocess
args = ["echo", 'a=?',"read", "a", "echo", "a=$a"]
subprocess.call(args)
and when I execute it, I have in terminal this report:
Armonicus#MyMacs-iMac MyNewFolder % python test.py
a=? read a echo a=$a
which is not at least close to what I expect.
Can I have some support from anyone, please?
There are a couple of issues with your approach here.
First, if what you're trying to do is prompt the user for input from the command line, then you can use Python builtins instead of a subprocess:
a = input('a=?')
print(a)
If you do want to call a subprocess with multiple commands, you need to either make separate calls for each command, or invoke a shell and execute the commands within it. For example:
subprocess.call("echo 'a=?'; read a; echo $a", shell=True)
I am calling this piece of code, but it produces some output in the console where I ran the python script (due to tee command):
os.system("echo 3 | sudo tee /proc/sys/vm/drop_caches")
This version does not produce console output but is there another way?
os.system('sudo bash -c "echo 3 > /proc/sys/vm/drop_caches"')
To answer the question based on its title in the most generic form:
To suppress all output from os.system(), append >/dev/null 2>&1 to the shell command, which silences both stdout and stderr; e.g.:
import os
os.system('echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null 2>&1')
Note that os.system() by design passes output from the calling process' stdout and stderr streams through to the console (terminal) - your Python code never sees them.
Also, os.system() does not raise an exception if the shell command fails and instead returns an exit code; note that it takes additional work to extract the shell command's true exit code: you need to extract the high byte from the 16-bit value returned, by applying >> 8 (although you can rely on a return value other than 0 implying an error condition).
Given the above limitations of os.system(), it is generally worthwhile to use the functions in the subprocess module instead:
For instance, subprocess.check_output() could be used as follows:
import subprocess
subprocess.check_output('echo 3 | sudo tee /proc/sys/vm/drop_caches', shell=True)
The above will:
capture stdout output and return it (with the return value being ignored in the example above)
pass stderr output through; passing stderr=subprocess.STDOUT as an additional argument would also capture stderr.
raise an error, if the shell command fails.
Note: Python 3.5 introduced subprocess.run(), a more flexible successor to both os.system() and subprocess.check_output() - see https://docs.python.org/3.5/library/subprocess.html#using-the-subprocess-module
Note:
The reason that the OP is employing tee in the first place - despite not being interested in stdout output - is that a naïve attempt to use > ... instead would be interpreted before sudo is invoked, and thus fail, because the required privileges to write to /proc/sys/... haven't been granted yet.
Whether you're using os.system() or a subprocess function, stdin is not affected by default, so if you're invoking your script from a terminal, you'll get an interactive password prompt when the sudo command is encountered (unless the credentials have been cached).
Write directly to the proc pseudo file instead via Python i/o lib.
This will require your script to run as root (via sudo), which means you should limit its scope to being an admin only tool. This also allows the script to run on boxes where sudo requires a password.
Example:
with open("/proc/sys/vm/drop_caches", "w") as drop_caches:
drop_caches.write("3")
subprocess.check_call(command,stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
you forgot to add stderr.
You also can use this simple method of subprocess module.
command = 'echo 3 | sudo tee /proc/sys/vm/drop_caches'
subprocess.check_call(shlex.split(command),stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL)
All outputs will be passed to DEVNULL. Any issues with the command will be reported by an exception. No issues means no output.
I believe, the easiest way to hide the console output when it's not possible with os.system is using os.popen:
os.popen("echo 3 | sudo tee /proc/sys/vm/drop_caches")
I'm trying to write a script that opens a new terminal then runs a separate python script from that terminal.
I've tried:
os.system("gnome-terminal 'python f.py'")
and
p = Popen("/usr/bin/gnome-terminal", stdin=PIPE)
p.communicate("python f.py")
but both methods only open a new terminal and do not run f.py. How would I go about opening the terminal AND running a separate script?
Edit:
I would like to open a new terminal window because f.py is a simply server that is running serve_forever(). I'd like the original terminal window to stay "free" to run other commands.
Like most terminals, gnome terminal needs options to execute commands:
gnome-terminal [-e, --command=STRING] [-x, --execute]
You probably need to add -x option:
x, --execute
Execute the remainder of the command line inside the terminal.
so:
os.system("gnome-terminal -x python f.py")
That would not run your process in the background unless you add & to your command line BTW.
The communicate attempt would need a newline for your input but should work too, but complex processes like terminals don't "like" being redirected. It seems like using an interactive tool backwards.
And again, that would block until termination. What could work would be to use p.stdin.write("python f.py\n") to give control to the python script. But in that case it's unlikely to work.
So it seems that you don't even need python do to what you want. You just need to run
python f.py &
in a shell.
As of GNOME Terminal 3.24.2 Using VTE version 0.48.4 +GNUTLS -PCRE2
Option “-x” is deprecated and might be removed in a later version of gnome-terminal.
Use “-- ” to terminate the options and put the command line to execute after it.
Thus the preferred syntax appears to be
gnome-terminal -- echo hello
rather than
gnome-terminal -x echo hello
Here is a complete example of how you would call a executable python file with subprocess.call Using argparse to properly parse the input.
the target process will print your given input.
Your python file to be called:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--file", help="Just A test", dest='myfile')
args = parser.parse_args()
print args.myfile
Your calling python file:
from subprocess import call
#call(["python","/users/dev/python/sandboxArgParse.py", "--file", "abcd.txt"])
call(["gnome-terminal", "-e", "python /users/dev/python/sandboxArgParse.py --file abcd.txt"])
Just for information:
You probably don't need python calling another python script to run a terminal window with a process, but could do as follows:
gnome-terminal -e "python /yourfile.py -f yourTestfile.txt"
The following code will open a new terminal and execute the process:
process = subprocess.Popen(
"sudo gnome-terminal -x python f.py",
stdout=subprocess.PIPE,
stderr=None,
shell=True
)
I am running a uWS server with this.In my case Popen didn't help(Even though it run the executable, still it couldn't communicate with a client -: socket connection is broken).This is working.Also now they recommends to use "--" instead of "-e".
subprocess.call(['gnome-terminal', "--", "python3", "server_deployment.py"])
#server_deployment.py
def run():
execution_cmd = "./my_executable arg1 arg2 dll_1 dll_2"
os.system(execution_cmd)
run()
I am calling this piece of code, but it produces some output in the console where I ran the python script (due to tee command):
os.system("echo 3 | sudo tee /proc/sys/vm/drop_caches")
This version does not produce console output but is there another way?
os.system('sudo bash -c "echo 3 > /proc/sys/vm/drop_caches"')
To answer the question based on its title in the most generic form:
To suppress all output from os.system(), append >/dev/null 2>&1 to the shell command, which silences both stdout and stderr; e.g.:
import os
os.system('echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null 2>&1')
Note that os.system() by design passes output from the calling process' stdout and stderr streams through to the console (terminal) - your Python code never sees them.
Also, os.system() does not raise an exception if the shell command fails and instead returns an exit code; note that it takes additional work to extract the shell command's true exit code: you need to extract the high byte from the 16-bit value returned, by applying >> 8 (although you can rely on a return value other than 0 implying an error condition).
Given the above limitations of os.system(), it is generally worthwhile to use the functions in the subprocess module instead:
For instance, subprocess.check_output() could be used as follows:
import subprocess
subprocess.check_output('echo 3 | sudo tee /proc/sys/vm/drop_caches', shell=True)
The above will:
capture stdout output and return it (with the return value being ignored in the example above)
pass stderr output through; passing stderr=subprocess.STDOUT as an additional argument would also capture stderr.
raise an error, if the shell command fails.
Note: Python 3.5 introduced subprocess.run(), a more flexible successor to both os.system() and subprocess.check_output() - see https://docs.python.org/3.5/library/subprocess.html#using-the-subprocess-module
Note:
The reason that the OP is employing tee in the first place - despite not being interested in stdout output - is that a naïve attempt to use > ... instead would be interpreted before sudo is invoked, and thus fail, because the required privileges to write to /proc/sys/... haven't been granted yet.
Whether you're using os.system() or a subprocess function, stdin is not affected by default, so if you're invoking your script from a terminal, you'll get an interactive password prompt when the sudo command is encountered (unless the credentials have been cached).
Write directly to the proc pseudo file instead via Python i/o lib.
This will require your script to run as root (via sudo), which means you should limit its scope to being an admin only tool. This also allows the script to run on boxes where sudo requires a password.
Example:
with open("/proc/sys/vm/drop_caches", "w") as drop_caches:
drop_caches.write("3")
subprocess.check_call(command,stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
you forgot to add stderr.
You also can use this simple method of subprocess module.
command = 'echo 3 | sudo tee /proc/sys/vm/drop_caches'
subprocess.check_call(shlex.split(command),stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL)
All outputs will be passed to DEVNULL. Any issues with the command will be reported by an exception. No issues means no output.
I believe, the easiest way to hide the console output when it's not possible with os.system is using os.popen:
os.popen("echo 3 | sudo tee /proc/sys/vm/drop_caches")
I am writing a small python script that needs to execute git commands from inside a given directory
The code is as follows:
import subprocess, os
pr = subprocess.Popen(['/usr/bin/git', 'status'],
cwd=os.path.dirname('/path/to/dir/'),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True)
(out, error) = pr.communicate()
print out
But it shows git usage as the output.
If the command doesn't involve git for eg. ['ls'] then it shows the correct output.
Is there anything I am missing ?
python version - 2.6.6
Thanks.
subprocess.Popen:
On Unix, with shell=True: […] If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself.
You don't want shell=True and also a list of arguments. Set shell=False.