When I run the command
subprocess.call(['intersectBed','-u','-a',out_snv_filter,'-b',cds,'>',out_cds],shell=True)
I get the help menu for intersectBed reported back in the interpreter.
But when I run
>>> ' '.join(['intersectBed','-u','-a',out_snv_filter,'-b',cds,'>',out_cds])
'intersectBed -u -a test/test.out.snv.filter -b gencode7.cds.bed > test/test.out.cds'
$ intersectBed -u -a test/test.out.snv.filter -b gencode7.cds.bed > test/test.out.cds
The program runs normally. What is the difference here?
from subprocess import check_call
args = ['intersectBed','-u','-a',out_snv_filter,'-b',cds]
with open(out_cds, 'wb') as outfile:
check_call(args, stdout=outfile)
Related
What is the best way to execute the below command in Python in a single line?
echo $(readlink /sys/dev/block/$(mountpoint -d /))
Tried using individual os.system(cmd) by separating - "mountpoint -d /" first and taking the output and appending to "readlink /sys/dev/block/${0}".format(out.strip()) and doing an echo works. Tried using subprocess and subprocess.Popen and subprocess.check_output but it raises raise CalledProcessError
cmd = "echo $(readlink /sys/dev/block/$(mountpoint -d /))"
You have to call the subcommand separately. And you can use python methods to read the link:
import subprocess
import os
path = "/"
device = subprocess.run(["mountpoint", "-d", path], stdout=subprocess.PIPE, encoding="utf8").stdout.strip()
link = os.readlink("/sys/dev/block/" + device)
print(link)
You probably want to use something like the following:
cmd = "bash -c 'echo $(readlink /sys/dev/block/$(mountpoint -d /))'"
echo doesn't substitute $() blocks, that's what your shell does, so you have to call the shell. os.system(cmd) should work then.
I am trying to port:
https://coderwall.com/p/ewk0mq/stop-remove-all-docker-containers
to a python script. So far I have:
def remove_all_containers():
subprocess.call(['docker', 'stop','$(docker ps -a -q)'])
subprocess.call(['docker', 'rm','$(docker ps -a -q)'])
return;
But get:
Error response from daemon: No such container: $(docker ps -a -q)
I have also tried:
def remove_all_containers():
subprocess.call(['docker', 'stop',$(docker ps -a -q)])
subprocess.call(['docker', 'rm',$(docker ps -a -q)])
return;
But that gives:
subprocess.call(['docker', 'stop',$(docker ps -a -q)])
SyntaxError: invalid syntax
it seems I need to nest another subprocess call into the parent subprocess call. Or is there a simpler way to do this?
TL;DR: Command substitution $(...) is a shell feature, therefore you must run your commands on a shell:
subprocess.call('docker stop $(docker ps -a -q)', shell=True)
subprocess.call('docker rm $(docker ps -a -q)', shell=True)
Additional improvements:
It's not required, but I would suggest using check_call (or run(..., check=True), see below) instead of call(), so that if an error occurs it doesn't go unnoticed:
subprocess.check_call('docker stop $(docker ps -a -q)', shell=True)
subprocess.check_call('docker rm $(docker ps -a -q)', shell=True)
You can also go another route: parse the output of docker ps -a -q and then pass to stop and rm:
container_ids = subprocess.check_output(['docker', 'ps', '-aq'], encoding='ascii')
container_ids = container_ids.strip().split()
if container_ids:
subprocess.check_call(['docker', 'stop'] + container_ids])
subprocess.check_call(['docker', 'rm'] + container_ids])
If you're using Python 3.5+, you can also use the newer run() function:
# With shell
subprocess.run('docker stop $(docker ps -a -q)', shell=True, check=True)
subprocess.run('docker rm $(docker ps -a -q)', shell=True, check=True)
# Without shell
proc = subprocess.run(['docker', 'ps', '-aq'], check=True, stdout=PIPE, encoding='ascii')
container_ids = proc.stdout.strip().split()
if container_ids:
subprocess.run(['docker', 'stop'] + container_ids], check=True)
subprocess.run(['docker', 'rm'] + container_ids], check=True)
There is nice official library for python, that helps with Docker.
https://docker-py.readthedocs.io/en/stable/index.html
import docker
client = docker.DockerClient(Config.DOCKER_BASE_URL)
docker_containers = client.containers.list(all=True)
for dc in docker_containers:
dc.remove(force=True)
We've received all containers and remove them all doesn't matter container status is 'started' or not.
The library could be useful if you can import it into code.
I have a ssh command which I was using in a system() statement but I want to replace it with a subprocess.call() statement. My ssh command is:
cmd ="ssh -i pem-file.pem user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"
system(cmd)
I want to replace above with a subprocess.call() statement as it is giving me some performance issues and I read that subprocess.call() is a much better option to use. I formulated this query but it is not executing:
result = subprocess.call(["ssh","-i", "pem-file.pem","user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"])
What is the mistake I am doing and what is the correct syntax?
The function shlex.split() is useful for parsing command line arguments into the proper format. This should resolve your syntax error:
import shlex
import subprocess
cmd ="ssh -i pem-file.pem user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"
result = subprocess.call(shlex.split(cmd))
If that doesn't fix your error, then you can pass subprocess.call the shell=True argument:
import subprocess
cmd ="ssh -i pem-file.pem user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"
result = subprocess.call(cmd, shell=True)
Using the shell argument will cause your command to be executed through a shell, rather than having the interpreter parse it. However, don't use the shell option if cmd can ever come from an untrusted source. Take at look at the warning in the Python docs.
One more note:
subprocess.system() is newer and more flexible than os.system(), but don't worry too much about "upgrading" to the new function. The advantages of subprocess.call() are in the more flexible options for communicating with your subprocess. If all you're doing is executing a single command and getting the return code, os.system() is probably fine. If you're finding that your command is being flaky and unreliable, switching to subprocess.call() probably isn't going to help much.
Assuming there are no shell meta-characters in hostname (likely), your command could look like this: each command-line argument is a separate list item:
#!/usr/bin/env python
import subprocess
cmd = ["ssh", "-i", "pem-file.pem", "user#" + hostname,
"cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 "
"-f text-file_" + ts + ".txt"]
subprocess.check_call(cmd)
Unlike os.system(); it doesn't run the (local) shell.
You could get the argument list using shlex.split(your_original_system_command) (mentioned by #skrrgwasme) but shlex.split() can be fooled and therefore it is mostly useful as a hint on how the result should look like.
I have the following command that works in the shell:
$ pv itunes20140910.tbz | sudo tar xpj -C /tmp
However, when I try and do it in python, it doesn't work:
>>> import subprocess
>>> import shlex
>>> cmd=shlex.split('pv itunes20140910.tbz | sudo tar xpj -C /tmp')
>>> subprocess.call(cmd)
pv: invalid option -- 'C'
Try `pv --help' for more information.
1
What am I doing wrong here, and what would be the correct command to run in python?
The above answers didn't have the net effect of what I was looking for (the progress bar), though the command would run without error. Here is what worked for me:
>>> import shlex, subprocess
>>> p1 = subprocess.Popen(shlex.split('pv /tmp/itunes20140910.tbz'), stdout=subprocess.PIPE) #Set up the echo command and direct the output to a pipe
>>> subprocess.Popen(shlex.split('sudo tar xpj -C /tmp'), stdin=p1.stdout) #send p1's output to p2
Use shell=True argument. Otherwise | cannot be interpreted.
subprocess.call('pv itunes20140910.tbz | sudo tar xpj -C /tmp', shell=True)
I'm trying to execute a rsync command via subrocess & popen. Everything's ok until I don't put the rsh subcommand where things go wrong.
from subprocess import Popen
args = ['-avz', '--rsh="ssh -C -p 22 -i /home/bond/.ssh/test"', 'bond#localhost:/home/bond/Bureau', '/home/bond/data/user/bond/backups/']
p = Popen(['rsync'] + args, shell=False)
print p.wait()
#just printing generated command:
print ' '.join(['rsync']+args)
I've tried to escape the '--rsh="ssh -C -p 22 -i /home/bond/.ssh/test"' in many ways, but it seems that it's not the problem.
I'm getting the error
rsync: Failed to exec ssh -C -p 22 -i /home/bond/.ssh/test: No such file or directory (2)
If I copy/paste the same args that I output at the time, I'm getting a correct execution of the command.
Thanks.
What happens if you use '--rsh=ssh -C -p 22 -i /home/bond/.ssh/test' instead (I removed the double quotes).
I suspect that this should work. What happens when you cut/paste your line into the commandline is that your shell sees the double quotes and removes them but uses them to prevent -C -p etc. from being interpreted as separate arguments. when you call subprocess.Popen with a list, you've already partitioned the arguments without the help of the shell, so you no longer need the quotes to preserve where the arguments should be split.
Having the same problem, I googled this issue extensively. It would seem you simply cannot pass arguments to ssh with subprocess. Ultimately, I wrote a shell script to run the rsync command, which I could pass arguments to via subprocess.call(['rsyncscript', src, dest, sshkey]). The shell script was: /usr/bin/rsync -az -e "ssh -i $3" $1 $2
This fixed the problem.