How do I NOT parallelize local commands inside a task with #parallel decorator:
#parallel
def myTask():
local('initialize localhost')
run('command on remote host')
local('clean up localhost')
I want commands on local host only to execute once, and commands for remote hosts run in parallel. Right now I'm seeing local host commands running for each remote host instance. What is the cleanest way to do this?
Thanks
Group your parallel commands into a decorated function. Then use execute() to call it inbetween the local calls.
Does the following work for you?
def local_init():
local('initialize')
#parallel
def myTask():
run('remote command')
def local_cleanup():
local('clean up')
And then:
fab local_init myTask local_cleanup
Related
In fabric2's task definition, how do I get the command line argument "-H --hosts"? Since I need to create connection myself.
eg.
fab -H web1,db1 task1
#task
def task1(c):
# HOW TO GET 'web1,db1' HERE???
with Connection(host=???, connect_kwargs={}) as conn:
conn.put('a','b')
do_my_stuff()
After a short while I figured it out.
#task
def task1(ctx):
with Connection(host=ctx.host, connect_kwargs={}) as c:
c.put('a', '/tmp/a')
Here is my script :
from fabric2 import Connection
c = Connection('127.0.0.1')
with c.cd('/home/bussiere/'):
c.run('ls -l')
But i have this error :
paramiko.ssh_exception.AuthenticationException: Authentication failed.
So how do I run a command on localhost ?
In Fabric2, the Connection object has got a local() method.
Have a look at this object's documentation here.
As of July 2020, with fabric2 if you don't pass argument to your task decorator by default you are on the local machine.
for example the following will run on your local machine (localhost) :
Example 1 : Only on local
#python3
#fabfile.py
from fabric import task, Connection
c = Connection('remote_user#remote_server.com')
#task
def DetailList(c):
c.run('ls -l') # will run on local server because the decorator #task does not contain the hosts parameter
You then would run this on your machine with
fab DetailList
If you want to mix code that should be running on remote server and on local you should pass the connection to the #task decorator as a parameter.
Example 2: on local and on remote (but different functions)
#python3
#fabfile.py
#imports
from fabric import task, Connection
#variables
list_of_hosts = ['user#yourserver.com'] #you should have already configure the ssh access
c = Connection(list_of_hosts[0])
working_dir = '/var/www/yourproject'
#will run on remote
#task(hosts = list_of_hosts)
def Update(c):
c.run('sudo apt get update') # will run on remote server because hosts are passed to the task decorator
c.run(f'cd {working_dir} && git pull') # will run on remote server because hosts are passed to the task decorator
c.run('sudo service apache2 restart') # will run on remote server because hosts are passed to the task decorator
#will run on local because you do not specify a host
#task
def DetailsList(c):
c.run('ls -l') # # will run on local server because hosts are NOT passed to the task decorator
As mentionned by Ismaïl there also is a 'local' method that can be used when passing the hosts parameter, the 'local' method will run on the localhost although you have specified the host parameter to the task decorator. Be careful though you can not use the 'local' method if you didn't specified any hosts parameters, use run instead as shown in example 1 & 2.
Example 3 : use both on remote and local servers but under the same functions, note we are not decorating functions that are called in the UpdateAndRestart function.
#python3
#fabfile.py
#imports
from fabric import task, Connection
#variables
list_of_hosts = ['www.yourserver.com'] #you should have already configure the ssh access
c = Connection(list_of_hosts[0])
working_dir = '/var/www/yourproject'
def UpdateServer(c):
c.run('sudo apt get update') # will run on remote server because hosts are passed to the task decorator
c.local('echo the remote server is now updated') # will run on local server because you used the local method when hosts are being passed to the decorator
def PullFromGit(c):
c.run(f'cd {working_dir} && git pull') # will run on remote server because hosts are passed to the task decorator
c.local('echo Git repo is now pulled') # will run on local server because you used the local method when hosts are being passed to the decorator
def RestartServer(c):
c.run('sudo service apache2 restart') # will run on remote server because hosts are passed to the task decorator
c.local('echo Apache2 is now restarted') # will run on local server because you used the local method when hosts are being passed to the decorator
#task(hosts = list_of_hosts)
def UpdateAndRestart(c):
UpdateServer(c)
PullFromGit(c)
RestartServer(c)
c.local('echo you have updated, pulled and restarted Apache2') # will run on local server because you used the local method when hosts are being passed to the decorator
You will be able to run the entire stack with :
fab UpdateAndRestart
Is it possible to have local and remote tasks execute from within the same task method?
e.g., I want to do something like the following:
#fabric.api.task
def Deploy():
PrepareDeploy()
PushDeploy()
execute(Extract())
execute(Start())
Where PrepareDeploy and PushDeploy are local tasks (executing only locally, via the fabric.api.local() method):
#fabric.api.task
#fabric.decorators.runs_once
def PrepareDeploy():
#fabric.api.task
#fabric.decorators.runs_once
def PushDeploy():
And Extract/Start are methods that should be run on the remote hosts themselves:
#fabric.api.task
def Extract():
#fabric.api.task
def Start():
However, when I try to do fab Deploy, I get something like:
[remote1.serv.com] Executing task 'Deploy'
[localhost] local: find . -name "*.java" > sources.txt
...
The first line seems wrong to me (and in fact, causes errors).
You can spawn new task and defining on what hosts should it run, for example - how to create rabbitmq of all hosts are provisioned with puppet with the same erlang cookie.
See around line 114 - there is an executon of the tasks on specific hosts.
https://gist.github.com/nvtkaszpir/17d2e2180771abd93c46
I hope this helps.
I'm attempting to announce deployment start and end in my fabric script. Feels like this should be easy, but for the life of me I can't figure out how to do it.
env.hosts = ['www1', 'www2', 'www3', 'www4']
def announce_start():
# code to connect to irc server and announce deployment begins
pass
def announce_finish():
# code to connect to irc server and announce deployment finishes
pass
def deploy():
# actual deployment code here
pass
Here's what I've tried:
If I make my deploy task contain 'announce_start' and 'announce_finish'. It will attempt to run all those tasks on each server.
def deploy():
announce_start()
# actual deployment code here
announce_finish()
If I decorate announce_start() and announce_end() with #hosts('localhost'), it runs it on localhost, but still four times. One for each host.
As I was typing this, I finally got it to work by using the decorator #hosts('localhost') on announce_start/end and the fab command:
fab announce_start deploy announce_end
But this seems a bit hacky. I'd like it all wrapped in a single deploy command. Is there a way to do this?
You can use fabric.api.execute, e.g.
def announce_start():
# code to connect to irc server and announce deployment begins
pass
def announce_finish():
# code to connect to irc server and announce deployment finishes
pass
#hosts(...)
def deploy_machine1():
pass
#hosts(...)
def deploy_machine2():
pass
def deploy():
announce_start()
execute(deploy_machine1)
execute(deploy_machine2)
announce_finish()
and then just invoke fab deploy
#import ssh
import socket
from fabric.operations import run
def connect_and_wait():
#ssh.config.socket.setdefaulttimeout(5)
socket.setdefaulttimeout(5)
print('SSTART')
run('echo START')
run('sleep 10')
run('echo END')
print('EEND')
The script above prints everything without any error/exception.
Python 2.6.5, Fabric 1.4.2.
socket.setdefaulttimeout() does not work.
ssh.config.socket.setdefaulttimeout() does not work.
fabric.api.env['timeout'] is for connecting phase only I suppose.
Fabric uses "lazy" connections to remote hosts and can automatically reconnect when executing task on a host and connection is lost. Seems there is no way to explicitly drop idling connections, but you can close all connections and let fabric reconnect to "active" hosts. fabric.network.disconnect_all() do the trick.