I would like to pass a few values from fabric into the remote environment, and I'm not seeing a great way to do it. The best I've come up with so far is:
with prefix('export FOO=BAR'):
run('env | grep BAR')
This does seem to work, but it seems like a bit of a hack.
I looked in the GIT repository and it looks like this is issue #263.
As of fabric 1.5 (released), fabric.context_managers.shell_env does what you want.
with shell_env(FOO1='BAR1', FOO2='BAR2', FOO3='BAR3'):
local("echo FOO1 is $FOO1")
I think your prefix-based solution is perfectly valid. However, if you want to have a shell_env context manager as the one proposed in issue#263, you can use the following alternative implementation in your fab files:
from fabric.api import run, env, prefix
from contextlib import contextmanager
#contextmanager
def shell_env(**env_vars):
orig_shell = env['shell']
env_vars_str = ' '.join('{0}={1}'.format(key, value)
for key, value in env_vars.items())
env['shell']='{0} {1}'.format(env_vars_str, orig_shell)
yield
env['shell']= orig_shell
def my_task():
with prefix('echo FOO1=$FOO1, FOO2=$FOO2, FOO3=$FOO3'):
with shell_env(FOO1='BAR1', FOO2='BAR2', FOO3='BAR3'):
run('env | grep BAR')
Note that this context manager modifies env['shell'] instead of env['command_prefixes'] (as prefix context manager does), so you:
can still use prefix (see example output below) without the interaction problems mentioned in issue#263.
have to apply any changes to env['shell'] before using shell_env. Otherwise, shell_env changes will be overwritten and environment variables won't be available for your commands.
When executing the fab file above, you get the following output:
$ fab -H localhost my_task
[localhost] Executing task 'my_task'
[localhost] run: env | grep BAR
[localhost] out: FOO1=BAR1, FOO2=BAR2, FOO3=BAR3
[localhost] out: FOO1=BAR1
[localhost] out: FOO2=BAR2
[localhost] out: FOO3=BAR3
[localhost] out:
Done.
Disconnecting from localhost... done.
Fabric 1.5.0 (currently in Git) takes shell as local() named argument.
If you pass '/bin/bash' there it passes it to executable argument of Popen.
It won't execute your .bashrc though because .bashrc is sourced on interactive invocation of bash. You can source any file you want inside local:
local('. /usr/local/bin/virtualenvwrapper.sh && workon focus_tests && bunch local output', shell='/bin/bash')
Another way is to pass a value through command line with --set:
--set=domain=stackoverflow.com
Then, you can address to it in script with env.domain
see http://docs.fabfile.org/en/1.11/usage/fab.html#cmdoption--set
Try using decorator
from fabric.context_managers import shell_env
from functools import wraps
def set_env():
def decorator(func):
#wraps(func)
def inner(*args, **kwargs):
with shell_env(DJANGO_CONFIGURATION=env.config):
run("echo $DJANGO_CONFIGURATION")
return func(*args, **kwargs)
return inner
return decorator
#task
#set_env()
def testme():
pass
Related
I am using the subprocess.call to execute a shell script from another application I am integrating with. This script sets environment variables with export MY_VAR=foo. Next, I need to execute more commands over subprocess with the environment that was set by the shell script.
How to extract the state of environment from the child process? It only returns the errno code.
i.e. I want to run:
subprocess.call(["export", "MY_VAR=foo"]
subprocess.call(["echo", "$MY_VAR"]) # should print 'foo'.
I know that I can set environment with env keyword, but the point of my question is how to get the environment variables that a subprocess sets. In shell you can source any script to get it's declared environment variables. What's the alternative in python?
I ran into this issue just recently. It seems that this is a difficult problem for reasons upstream of Python: posix_spawn doesn't give a way to read the environment variables of the spawned process, nor is there any easy way to read the environment of a running process.
Bash's source is specific to running bash code in the bash interpreter: it just evals the file in the current bash interpreter rather than starting a subprocess. This mechanism can't work if you are running bash code from Python.
It is possible to make a separate mechanism specific to running bash code from Python. The following is the best that I could manage. Would be nice to have a less flimsy solution.
import json
import os
import subprocess
import sys
from contextlib import AbstractContextManager
class BashRunnerWithSharedEnvironment(AbstractContextManager):
"""Run multiple bash scripts with persisent environment.
Environment is stored to "env" member between runs. This can be updated
directly to adjust the environment, or read to get variables.
"""
def __init__(self, env=None):
if env is None:
env = dict(os.environ)
self.env: Dict[str, str] = env
self._fd_read, self._fd_write = os.pipe()
def run(self, cmd, **opts):
if self._fd_read is None:
raise RuntimeError("BashRunner is already closed")
write_env_pycode = ";".join(
[
"import os",
"import json",
f"os.write({self._fd_write}, json.dumps(dict(os.environ)).encode())",
]
)
write_env_shell_cmd = f"{sys.executable} -c '{write_env_pycode}'"
cmd += "\n" + write_env_shell_cmd
result = subprocess.run(
["bash", "-ce", cmd], pass_fds=[self._fd_write], env=self.env, **opts
)
self.env = json.loads(os.read(self._fd_read, 5000).decode())
return result
def __exit__(self, exc_type, exc_value, traceback):
if self._fd_read:
os.close(self._fd_read)
os.close(self._fd_write)
self._fd_read = None
self._fd_write = None
def __del__(self):
self.__exit__(None, None, None)
Example:
with BashRunnerWithSharedEnvironment() as bash_runner:
bash_runner.env.pop("A", None)
res = bash_runner.run("A=6; echo $A", stdout=subprocess.PIPE)
assert res.stdout == b'6\n'
assert bash_runner.env.get("A", None) is None
bash_runner.run("export A=2")
assert bash_runner.env["A"] == "2"
res = bash_runner.run("echo $A", stdout=subprocess.PIPE)
assert res.stdout == b'2\n'
res = bash_runner.run("A=6; echo $A", stdout=subprocess.PIPE)
assert res.stdout == b'6\n'
assert bash_runner.env.get("A", None) == "6"
bash_runner.env["A"] = "7"
res = bash_runner.run("echo $A", stdout=subprocess.PIPE)
assert res.stdout == b'7\n'
assert bash_runner.env["A"] == "7"
It is not possible, because the environment is changed only in the child process. You might from there return it as output to STDOUT, STDERR - but as soon as the subprocess is terminated, You can not access anything from it.
# this is process #1
subprocess.call(["export", "MY_VAR=foo"]
# this is process #2 - it can not see the environment of process #1
subprocess.call(["echo", "$MY_VAR"]) # should print 'foo'.
Not sure I see the problem here. You just need to remember the following:
each subprocess that gets started is independent of any setups done in previous subprocesses
if you want to set up some variables and use them, do both those things in ONE process
So make setupVars.sh like this:
export vHello="hello"
export vDate=$(date)
export vRandom=$RANDOM
And make printVars.sh like this:
#!/bin/bash
echo $vHello, $vDate, $vRandom
And make that executable with:
chmod +x printVars.sh
Now your Python looks like this:
import subprocess
subprocess.call(["bash","-c","source setupVars.sh; ./printVars.sh"])
Output
hello, Mon Jul 12 00:32:29 BST 2021, 8615
Is there any way to get this to work with env.hosts? As opposed to having to loop manually whenever I have multiple hosts to run this on?
I am trying to use the fabric api, to not have to use the very inconvenient and kludgey fabric command line call. I set the env.hosts variable in one module/class and then call a another class instance method to run a fabric command. In the called class instance I can print out the env.hosts list. Yet when I try to run a command it tells me it can't find a host.
If I loop through the env.hosts array and manually set the env.host variable for each host in the env.hosts array, I can get the run command to work. What is odd is that I also set the env.user variable in the calling class and it is picked up.
e.g. this works:
def upTest(self):
print('env.hosts = ' + str(env.hosts))
for host in env.hosts:
env.host_string = host
print('env.host_string = ' + env.host_string)
run("uptime")
output from this:
env.hosts = ['ec2-....amazonaws.com']
env.host_string = ec2-....amazonaws.com
[ec2-....amazonaws.com] run: uptime
[ec2-....amazonaws.com] out: 18:21:15 up 2 days, 2:13, 1 user, load average: 0.00, 0.01, 0.05
[ec2-....amazonaws.com] out:
This doesn't work... but it does work if you run it from a "fab" file... makes no sense to me.
def upTest(self):
print('env.hosts = ' + str(env.hosts))
run("uptime")
This is the output:
No hosts found. Please specify (single) host string for connection:
I did try putting an #task decorator on the method (and removing the 'self' reference since the decorator didn't like that). But to no help.
Is there any way to get this to work with env.hosts? As opposed to having to loop manually whenever I have multiple hosts to run this on?
Finally, I fixed this problem by using execute() and exec.
main.py
#!/usr/bin/env python
from demo import FabricSupport
hosts = ['localhost']
myfab = FabricSupport()
myfab.execute("df",hosts)
demo.py
#!/usr/bin/env python
from fabric.api import env, run, execute
class FabricSupport:
def __init__(self):
pass
def hostname(self):
run("hostname")
def df(self):
run("df -h")
def execute(self,task,hosts):
get_task = "task = self.%s" % task
exec get_task
execute(task,hosts=hosts)
python main.py
[localhost] Executing task 'hostname'
[localhost] run: hostname
[localhost] out: heydevops-workspace
I've found that it's best not to set env.hosts in code but instead to define roles based on your config file and use the fab tool to specify a role. It worked for me
my_roles.json
{
"web": [ "user#web1.example.com", "user#web2.example.com" ],
"db": [ "user#db1.example.com", "user#db2.example.com" ]
}
fabfile.py
from fabric.api import env, run, task
import json
def load_roles():
with open('my_roles.json') as f:
env.roledefs = json.load(f)
load_roles()
#task
def my_task():
run("hostname")
CLI
fab -R web my_task
output from running my_task for each of web1 and web2 is here
I can't seem to figure this one out but when I do a very simple test to localhost to have fabric execute this command run('history'), the resulting output on the command line is blank.
Nor will this work either: run('history > history_dump.log')
Here is the complete FabFile script below, obviously I'm missing something here.
-- FabFile.py
from fabric.api import run, env, hosts, roles, parallel, cd, task, settings, execute
from fabric.operations import local,put
deploymentType = "LOCAL"
if (deploymentType == "LOCAL"):
env.roledefs = {
'initial': ['127.0.0.1'],
'webservers': ['127.0.0.1'],
'dbservers' : ['127.0.0.1']
}
env.use_ssh_config = False
# Get History
# -------------------------------------------------------------------------------------
#task
#roles('initial')
def showHistoryCommands():
print("Logging into %s and accessing the command history " % env.host_string)
run('history') #does not display anything
run('history > history_dump.log') #does not write anything out
print "Completed displaying the command history"
Any suggestions/solutions would be most welcomed.
History is a shell builtin, so it doesn't work like a normal command. I think your best bet would be to try and read the history file from the filesystem.
local('cat ~/.bash_history')
or
run('cat ~/.bash_history')
Substitute for the appropriate history file path.
To expand a bit after some research, the command succeeds when run, but for some reason, be it that fabric neither captures or prints the output. Or the way history prints it's output. While other builtins commands like env work fine. So for now I don't know what exactly is going on.
I cannot get fabric working when used as a library within my own python scripts. I made a very short example fabfile.py to demonstrate my problem:
#!/usr/bin/env python
from fabric.api import *
print("Hello")
def test():
with settings(host_string='myIp', user="myUser", password="myPassword"):
run("hostname")
if __name__ == '__main__':
test()
Running fab works like a charm:
$ fab test
Hello
[myIp] run: hostname
[myIp] out: ThisHost
[myIp] out:
Done.
Disconnecting from myUser#myIp... done.
Ok, now, running the python script without fab seems to break somewhere:
$ python fabfile.py
Hello
[myIp] run: hostname
It immediatly returns, so it does not even seem to wait for a response. Maybe there are errors, but I don't see how to output those.
I am running this script inside my vagrant virtual machine. As fab executes without any errors, I guess this should not be a problem!
UPDATE
The script seems to crash as it does not execute anything after the first run. local on the other hand works!
We executed the script on a co-workers laptop and it runs without any issues. I am using Python 2.6.5 on Ubuntu 10.04 with fabric 1.5.1, so I guess there is a problem with some of this! Is there any way to debug this properly?
I've experienced a similar issue, that the fab command exited without error but just a blank line on the first run()/sudo() command.
So I put the run() command into a try: except: block and printed the traceback:
def do_something():
print(green("Executing on %(host)s as %(user)s" % env))
try:
run("uname -a")
except:
import traceback
tb = traceback.format_exc()
print(tb)
I saw that it the script exited in the fabfile/network.py at line 419 when it caught an EOFError or TypeError. I modified the script to:
...
except (EOFError, TypeError) as err:
print err
# Print a newline (in case user was sitting at prompt)
print('')
sys.exit(0)
...
which then printed out:
connect() got an unexpected keyword argument 'sock'
So I remove the sock keyword argument in the connect method a few lines above and it worked like charm. I guess it is a problem with a paramiko version, that does not allow the sock keyword.
Versions:
Python 2.7.3
Fabric >= 1.5.3
paramiko 1.10.0
if you look at the fab command it looks like this:
sys.exit(
load_entry_point('Fabric==1.4.3', 'console_scripts', 'fab')()
)
this means it looks for a block labeled console_scripts in a file called entry_points.txt in the Fabric package and executes the methods listed there, in this case fabric.main:main
when we look at this method we see argument parsing, interesting fabfile importing and then:
if fabfile:
docstring, callables, default = load_fabfile(fabfile)
state.commands.update(callables)
....
for name, args, kwargs, arg_hosts, arg_roles, arg_exclude_hosts in commands_to_run:
execute(
name,
hosts=arg_hosts,
roles=arg_roles,
exclude_hosts=arg_exclude_hosts,
*args, **kwargs
)
with some experimentation we can come up with something like:
from fabric import state
from fabric.api import *
from fabric.tasks import execute
from fabric.network import disconnect_all
def test():
with settings(host_string='host', user="user", password="password"):
print run("hostname")
if __name__ == '__main__':
state.commands.update({'test': test})
execute("test")
if state.output.status:
print("\nDone.")
disconnect_all()
which is obviously very incomplete, but perhaps you only need to add the
disconnect_all()
line at the end of your script
Say I have a fabfile.py that looks like this:
def setup():
pwd = getpass('mysql password: ')
run('mysql -umoo -p%s something' % pwd)
The output of this is:
[host] run: mysql -umoo -pTheActualPassword
Is there a way to make the output look like this?
[host] run: mysql -umoo -p*******
Note: This is not a mysql question!
Rather than modifying / overriding Fabric, you could replace stdout (or any iostream) with a filter.
Here's an example of overriding stdout to censor a specific password. It gets the password from Fabric's env.password variable, set by the -I argument. Note that you could do the same thing with a regular expression, so that you wouldn't have to specify the password in the filter.
I should also mention, this isn't the most efficient code in the world, but if you're using fabric you're likely gluing a couple things together and care more about manageability than speed.
#!/usr/bin/python
import sys
import string
from fabric.api import *
from fabric.tasks import *
from fabric.contrib import *
class StreamFilter(object):
def __init__(self, filter, stream):
self.stream = stream
self.filter = filter
def write(self,data):
data = data.replace(self.filter, '[[TOP SECRET]]')
self.stream.write(data)
self.stream.flush()
def flush(self):
self.stream.flush()
#task
def can_you_see_the_password():
sys.stdout = StreamFilter(env.password, sys.stdout)
print 'Hello there'
print 'My password is %s' % env.password
When run:
fab -I can_you_see_the_password
Initial value for env.password:
this will produce:
Hello there
My password is [[TOP SECRET]]
It may be better to put the password in the user's ~/.my.cnf under the [client] section. This way you don't have to put the password in the python file.
[client]
password=TheActualPassword
When you use the Fabric command run, Fabric isn't aware of whether or not the command you are running contains a plain-text password or not. Without modifying/overriding the Fabric source code, I don't think you can get the output that you want where the command being run is shown but the password is replaced with asterisks.
You could, however, change the Fabric output level, either for the entire Fabric script or a portion, so that the command being run is not displayed. While this will hide the password, the downside is that you wouldn't see the command at all.
Take a look at the Fabric documentation on Managing Output.
Write a shell script that invokes the command in question with the appropriate password, but without echoing that password. You can have the shell script lookup the password from a more secure location than from your .py files.
Then have fabric call the shell script instead.
This solves both the problem of having fabric not display the password and making sure you don't have credentials in your source code.
from fabric.api import run, settings
with settings(prompts={'Enter password: ': mysql_password}):
run("mysql -u {} -p -e {}".format(mysql_user,mysql_query))
or if no prompt available:
from fabric.api import run, hide
with hide('output','running','warnings'):
run("mycommand --password {}".format(my_password))