I am trying to port:
https://coderwall.com/p/ewk0mq/stop-remove-all-docker-containers
to a python script. So far I have:
def remove_all_containers():
subprocess.call(['docker', 'stop','$(docker ps -a -q)'])
subprocess.call(['docker', 'rm','$(docker ps -a -q)'])
return;
But get:
Error response from daemon: No such container: $(docker ps -a -q)
I have also tried:
def remove_all_containers():
subprocess.call(['docker', 'stop',$(docker ps -a -q)])
subprocess.call(['docker', 'rm',$(docker ps -a -q)])
return;
But that gives:
subprocess.call(['docker', 'stop',$(docker ps -a -q)])
SyntaxError: invalid syntax
it seems I need to nest another subprocess call into the parent subprocess call. Or is there a simpler way to do this?
TL;DR: Command substitution $(...) is a shell feature, therefore you must run your commands on a shell:
subprocess.call('docker stop $(docker ps -a -q)', shell=True)
subprocess.call('docker rm $(docker ps -a -q)', shell=True)
Additional improvements:
It's not required, but I would suggest using check_call (or run(..., check=True), see below) instead of call(), so that if an error occurs it doesn't go unnoticed:
subprocess.check_call('docker stop $(docker ps -a -q)', shell=True)
subprocess.check_call('docker rm $(docker ps -a -q)', shell=True)
You can also go another route: parse the output of docker ps -a -q and then pass to stop and rm:
container_ids = subprocess.check_output(['docker', 'ps', '-aq'], encoding='ascii')
container_ids = container_ids.strip().split()
if container_ids:
subprocess.check_call(['docker', 'stop'] + container_ids])
subprocess.check_call(['docker', 'rm'] + container_ids])
If you're using Python 3.5+, you can also use the newer run() function:
# With shell
subprocess.run('docker stop $(docker ps -a -q)', shell=True, check=True)
subprocess.run('docker rm $(docker ps -a -q)', shell=True, check=True)
# Without shell
proc = subprocess.run(['docker', 'ps', '-aq'], check=True, stdout=PIPE, encoding='ascii')
container_ids = proc.stdout.strip().split()
if container_ids:
subprocess.run(['docker', 'stop'] + container_ids], check=True)
subprocess.run(['docker', 'rm'] + container_ids], check=True)
There is nice official library for python, that helps with Docker.
https://docker-py.readthedocs.io/en/stable/index.html
import docker
client = docker.DockerClient(Config.DOCKER_BASE_URL)
docker_containers = client.containers.list(all=True)
for dc in docker_containers:
dc.remove(force=True)
We've received all containers and remove them all doesn't matter container status is 'started' or not.
The library could be useful if you can import it into code.
Related
What is the best way to execute the below command in Python in a single line?
echo $(readlink /sys/dev/block/$(mountpoint -d /))
Tried using individual os.system(cmd) by separating - "mountpoint -d /" first and taking the output and appending to "readlink /sys/dev/block/${0}".format(out.strip()) and doing an echo works. Tried using subprocess and subprocess.Popen and subprocess.check_output but it raises raise CalledProcessError
cmd = "echo $(readlink /sys/dev/block/$(mountpoint -d /))"
You have to call the subcommand separately. And you can use python methods to read the link:
import subprocess
import os
path = "/"
device = subprocess.run(["mountpoint", "-d", path], stdout=subprocess.PIPE, encoding="utf8").stdout.strip()
link = os.readlink("/sys/dev/block/" + device)
print(link)
You probably want to use something like the following:
cmd = "bash -c 'echo $(readlink /sys/dev/block/$(mountpoint -d /))'"
echo doesn't substitute $() blocks, that's what your shell does, so you have to call the shell. os.system(cmd) should work then.
I want to run a set of docker commands from python.
I tried creating a script like below and run the script from python using paramiko ssh_client to connect to the machine where the docker is running:
#!/bin/bash
# Get container ID
container_id="$(docker ps | grep hello | awk '{print $1}')"
docker exec -it $container_id sh -c "cd /var/opt/bin/ && echo $1 &&
echo $PWD && ./test.sh -q $1"
But docker exec ... never gets executed.
So I tried to run the below python script below, directly in the machine where the docker is running:
import subprocess
docker_run = "docker exec 7f34a9c1b78f /bin/bash -c \"cd
/var/opt/bin/ && ls -a\"".split()
subprocess.call(docker_run, shell=True)
I get a message: "Usage: docker COMMAND..."
But I get the expected results if I run the command
docker exec 7f34a9c1b78f /bin/bash -c "cd /var/opt/bin/ && ls -a"
directly in the machine
How to run multiple docker commands from the python script? Thanks!
You have a mistake in your call to subprocess.call. subprocess.call expects a command with a series of parameters. You've given it a list of parameter pieces.
This code:
docker_run = "docker exec 7f34a9c1b78f /bin/bash -c \"cd
/var/opt/bin/ && ls -a\"".split()
subprocess.call(docker_run, shell=True)
Runs this:
subprocess.call([
'docker', 'exec', '7f34a9c1b78f', '/bin/bash', '-c',
'"cd', '/var/opt/bin/', '&&', 'ls', '-a"'
], shell=True)
Instead, I believe you want:
subprocess.call([
'docker', 'exec', '7f34a9c1b78f', '/bin/bash', '-c',
'"cd /var/opt/bin/ && ls -a"' # Notice how this is only one argument.
], shell=True)
You might need to tweak that second call. I suspect you don't need the quotes ('cd /var/opt/bin/ && ls -a' might work instead of '"cd /var/opt/bin/ && ls -a"'), but I haven't tested it.
Following are a few methods worked:
Remove double quotes:
subprocess.call([
'docker', 'exec', '7f34a9c1b78f', '/bin/bash', '-c',
'cd /opt/teradata/tdqgm/bin/ && ./support-archive.sh -q 6b171e7a-7071-4975-a3ac-000000000241'
])
If you are not sure of how the command should be split up to pass it as an argument of subprocess method, shlex module:
https://docs.python.org/2.7/library/shlex.html#shlex.split
I have written a python script which includes this line:
response = subprocess.check_output(['/usr/bin/sudo /bin/su - backup -c "/usr/bin/ssh -q -o StrictHostKeyChecking=no %s bash -s" <<\'EOF\'\nPATH=/usr/local/bin:$PATH\nmvn --version|grep -i Apache|awk \'{print $3}\'|tr -d \'\n\'\nEOF' % i], shell=True)
This is in a for loop that goes through a list of hostnames and each one I want to check the result of the command on it. This works fine when I run it myself, however, this script is to be run by a system user (shinken - a nagios fork) and at that point I hit an issue.
shinken ALL=(ALL) NOPASSWD: ALL
However, I wanted to restrict the user to only allow it to run as the backup user:
shinken ALL=(backup) NOPASSWD: ALL
But when I run the script I get:
sudo: no tty present and no askpass program specified
I have read around this and tried a few things to fix it. I tried adding -t to my ssh command, but that didn't help. I believe I should be able to run the command with something similar to:
response = subprocess.check_output(['/usr/bin/sudo -u backup """ "/usr/bin/ssh -q -o StrictHostKeyChecking=no %s bash -s" <<\'EOF\'\nPATH=/usr/local/bin:$PATH\njava -version|grep -i version|awk \'{print $3}\'|tr -d \'\n\'\nEOF""" ' % i], shell=True)
But then I get this response:
subprocess.CalledProcessError: Command '['/usr/bin/sudo -u backup """ "/usr/bin/ssh -q -o StrictHostKeyChecking=no bamboo-agent-01 bash -s" <<\'EOF\'\nPATH=/usr/local/bin:$PATH\njava -version|grep -i version|awk \'{print $3}\'|tr -d \'\n\'\nEOF""" ']' returned non-zero exit status 1
If I run the command manually I get:
sudo: /usr/bin/ssh: command not found
Which is strange because that's where it lives.... I've no idea if what I'm trying is even possible. Thanks for any suggestions!
As for sudo:
shinken ALL=(backup) NOPASSWD: ALL
...only works when you switch directly from shinken to backup. You aren't doing that here. sudo su - backup is telling sudo to switch to root, and to run the command su - backup as root. Obviously, then, if you're going to use sudo su (which I've advised against elsewhere), you need your /etc/sudoers configuration to support that.
Because your /etc/sudoers isn't allowing direct the switch to root you're requesting, it's trying to prompt for a password, which requires a TTY, which is thus causing a failure.
Below, I'm rewriting the script to switch directly from shinken to backup, without going through root and running su:
As for the script:
import subprocess
remote_script='''
PATH=/usr/local/bin:$PATH
mvn --version 2>&1 | awk '/Apache/ { print $3 }'
'''
def maven_version_for_host(hostname):
# storing the command lets us pass it when constructing a CalledProcessError later
# could move it directly into the Popen creation if you don't need that.
cmd = [
'sudo', '-u', 'backup', '-i', '--',
'ssh', '-q', '-o', 'StrictHostKeyChecking=no', str(hostname),
'bash -s' # arguments in remote-command position to ssh all get concatenated
# together, so passing them as one command aids clarity.
]
proc = subprocess.Popen(cmd,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
response, error_string = proc.communicate(remote_script)
if proc.returncode != 0:
raise subprocess.CalledProcessError(proc.returncode, cmd, error_string)
return response.split('\n', 1)[0]
When I use docker run in interactive mode I am able to run the commands I want to test some python stuff.
root#pydock:~# docker run -i -t dockerfile/python /bin/bash
[ root#197306c1b256:/data ]$ python -c "print 'hi there'"
hi there
[ root#197306c1b256:/data ]$ exit
exit
root#pydock:~#
I want to automate this from python using the subprocess module so I wrote this:
run_this = "print('hi')"
random_name = ''.join(random.SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(20))
command = 'docker run -i -t --name="%s" dockerfile/python /bin/bash' % random_name
subprocess.call([command],shell=True,stderr=subprocess.STDOUT)
command = 'cat <<\'PYSTUFF\' | timeout 0.5 python | head -n 500000 \n%s\nPYSTUFF' % run_this
output = subprocess.check_output([command],shell=True,stderr=subprocess.STDOUT)
command = 'exit'
subprocess.call([command],shell=True,stderr=subprocess.STDOUT)
command = 'docker ps -a | grep "%s" | awk "{print $1}" | xargs --no-run-if-empty docker rm -f' % random_name
subprocess.call([command],shell=True,stderr=subprocess.STDOUT)
This is supposed to create the container, run the python command on the container and exit and remove the container. It does all this except the command is run on the host machine and not the docker container. I guess docker is switching shells or something like that. How do I run python subprocess from a new shell?
It looks like you are expecting the second command cat <<... to send input to the first command. But the two subprocess commands have nothing to do with each other, so this doesn't work.
Python's subprocess library, and the popen command that underlies it, offer a way to get a pipe to stdin of the process. This way, you can send in the commands you want directly from Python and don't have to attempt to get another subprocess to talk to it.
So, something like:
from subprocess import Popen, PIPE
p = Popen("docker run -i -t --name="%s" dockerfile/python /bin/bash", stdin=PIPE)
p.communicate("timeout 0.5 python | head -n 500000 \n" % run_this)
(I'm not a Python expert; apologies for errors in string-forming. Adapted from this answer)
You actually need to spawn a new child on the new shell you are opening.So after docker creation run docker enter or try the same operation with pexpect instead of subprocess.`pexpect spawns a new child and that way you can send commands.
When I run the following commands in Python interpreter, all is good, but fail when run as a script. Somehow the opened tunnel is not recognized. Any pointers will be appreciated.
Interpreter:
gateway = subprocess.Popen("ssh -N sshgw", shell=True)
scp_prod = "scp -r server:/NFS/{0} .".format(filePath)
subprocess.call(scp_prod, shell=True)
gateway.kill()
Script:
try:
gateway = subprocess.Popen("ssh -N sshgw", shell=True)
scp_prod = "scp -r server:/NFS/{0} .".format(filePath)
subprocess.call(scp_prod, shell=True)
finally:
gateway.kill()