Develop Reference

Unable to pass variable to a bash command in python

I am trying to pass a python variable to a bash command like this:
subscriptionId = "xxxxx"
command = " az account show -s $subscriptionId"
subprocess.check_output(command)
I get there following error:
error : az account show: error: argument --subscription/-s: expected one argument

Assigning a Python variable like subscriptionId = "xxxxx" does not magically place it in your environment, much less pass it to a subprocess. You need to do that interpolation yourself:
command = f"az account show -s {subscriptionId}"
If you really want to use environment variables, add the variable you want and enable shell expansion:
subscriptionId = ...
env = os.environ.copy()
env['subscriptionId'] = subscriptionId
command = "az account show -s ${subscriptionId}"
subprocess.check_output(command, env=env, shell=True)
Alternatively, you can mess with your own process environment:
subscriptionId = ...
os.environ['subscriptionId'] = subscriptionId
command = "az account show -s ${subscriptionId}"
subprocess.check_output(command, shell=True)
These options are, in my opinion, not recommended, since they raise all the security issues that shell=True brings with it, while providing you with no real advantage.

since the variable command is just a string you could simply do this.
subscriptionId = "xxxxx"
command = " az account show -s " + subscriptionId
subprocess.check_output(command)

Pytest Command Line: passing in string to use as host

So what I am trying to do is write a pytest script that allows me to pass in a string as an argument. I can't for the life of me figure out how to get the argument to pass in as a whole string instead of having it be treated as separate characters. I have tried a bunch of options but this is the closest I have gotten. I feel like I'm just missing some small logic that will make this actually work.
conftest.py
def pytest_addoption(parser):
parser.addoption("--host", dest="host", type="string",
action="store", default="john", help="Enter an
endpoint for TransactionDS")
def pytest_generate_tests(metafunc):
if 'host' in metafunc.fixturenames:
metafunc.parametrize("host", metafunc.config.getoption('host'))
test_args.py
def test_host(host):
global endpoint
endpoint = host
if (endpoint == '90.90.90.90'):
print(endpoint)
else:
print(endpoint)
command line
pytest -q --host "Chris" test_args.py -s --html=report.html
output
IP Address from Hello World 1 -
C
.h
.r
.i
.s
.

Fixture value supposed to be a list of values to iterate through.
So if you use string it will treat each char as a value and run the test the len(host) number of time
So just use
def pytest_generate_tests(metafunc):
if 'host' in metafunc.fixturenames:
metafunc.parametrize("host", [metafunc.config.getoption('host')])
py.test -q --host "Chris" test_args.py -s
Chris
.
1 passed in 0.00 seconds
I removed from your command line parameter html that is not described in your conftest.py

python fabric no host found must manually set 'env.host_string'

Is there any way to get this to work with env.hosts? As opposed to having to loop manually whenever I have multiple hosts to run this on?
I am trying to use the fabric api, to not have to use the very inconvenient and kludgey fabric command line call. I set the env.hosts variable in one module/class and then call a another class instance method to run a fabric command. In the called class instance I can print out the env.hosts list. Yet when I try to run a command it tells me it can't find a host.
If I loop through the env.hosts array and manually set the env.host variable for each host in the env.hosts array, I can get the run command to work. What is odd is that I also set the env.user variable in the calling class and it is picked up.
e.g. this works:
def upTest(self):
print('env.hosts = ' + str(env.hosts))
for host in env.hosts:
env.host_string = host
print('env.host_string = ' + env.host_string)
run("uptime")
output from this:
env.hosts = ['ec2-....amazonaws.com']
env.host_string = ec2-....amazonaws.com
[ec2-....amazonaws.com] run: uptime
[ec2-....amazonaws.com] out: 18:21:15 up 2 days, 2:13, 1 user, load average: 0.00, 0.01, 0.05
[ec2-....amazonaws.com] out:
This doesn't work... but it does work if you run it from a "fab" file... makes no sense to me.
def upTest(self):
print('env.hosts = ' + str(env.hosts))
run("uptime")
This is the output:
No hosts found. Please specify (single) host string for connection:
I did try putting an #task decorator on the method (and removing the 'self' reference since the decorator didn't like that). But to no help.
Is there any way to get this to work with env.hosts? As opposed to having to loop manually whenever I have multiple hosts to run this on?

Finally, I fixed this problem by using execute() and exec.
main.py
#!/usr/bin/env python
from demo import FabricSupport
hosts = ['localhost']
myfab = FabricSupport()
myfab.execute("df",hosts)
demo.py
#!/usr/bin/env python
from fabric.api import env, run, execute
class FabricSupport:
def __init__(self):
pass
def hostname(self):
run("hostname")
def df(self):
run("df -h")
def execute(self,task,hosts):
get_task = "task = self.%s" % task
exec get_task
execute(task,hosts=hosts)
python main.py
[localhost] Executing task 'hostname'
[localhost] run: hostname
[localhost] out: heydevops-workspace

I've found that it's best not to set env.hosts in code but instead to define roles based on your config file and use the fab tool to specify a role. It worked for me
my_roles.json
{
"web": [ "user#web1.example.com", "user#web2.example.com" ],
"db": [ "user#db1.example.com", "user#db2.example.com" ]
}
fabfile.py
from fabric.api import env, run, task
import json
def load_roles():
with open('my_roles.json') as f:
env.roledefs = json.load(f)
load_roles()
#task
def my_task():
run("hostname")
CLI
fab -R web my_task
output from running my_task for each of web1 and web2 is here

How to take command line input to a fabfile?

I want to give some input (initial configuration to a fabric file)
Something like:
fab deploy MyProject
where MyProject is the string I want to input. Is it possible to do so?

You will have to specify "MyProject" thus:
fab deploy:MyProject
And your function (in your fabfile.py) will look thus:
def deploy(project):
...
where project equals "MyProject".
(more info)

In your fabfile:
# fabfile.py
# ...
if not hasattr(env, 'my_setting'):
env.my_setting = 'default'
# ...
Run fab script:
fab --set=my_setting=my_value task
More info:
fab --help

Best way to add an environment variable in fabric?

I would like to pass a few values from fabric into the remote environment, and I'm not seeing a great way to do it. The best I've come up with so far is:
with prefix('export FOO=BAR'):
run('env | grep BAR')
This does seem to work, but it seems like a bit of a hack.
I looked in the GIT repository and it looks like this is issue #263.

As of fabric 1.5 (released), fabric.context_managers.shell_env does what you want.
with shell_env(FOO1='BAR1', FOO2='BAR2', FOO3='BAR3'):
local("echo FOO1 is $FOO1")

I think your prefix-based solution is perfectly valid. However, if you want to have a shell_env context manager as the one proposed in issue#263, you can use the following alternative implementation in your fab files:
from fabric.api import run, env, prefix
from contextlib import contextmanager
#contextmanager
def shell_env(**env_vars):
orig_shell = env['shell']
env_vars_str = ' '.join('{0}={1}'.format(key, value)
for key, value in env_vars.items())
env['shell']='{0} {1}'.format(env_vars_str, orig_shell)
yield
env['shell']= orig_shell
def my_task():
with prefix('echo FOO1=$FOO1, FOO2=$FOO2, FOO3=$FOO3'):
with shell_env(FOO1='BAR1', FOO2='BAR2', FOO3='BAR3'):
run('env | grep BAR')
Note that this context manager modifies env['shell'] instead of env['command_prefixes'] (as prefix context manager does), so you:
can still use prefix (see example output below) without the interaction problems mentioned in issue#263.
have to apply any changes to env['shell'] before using shell_env. Otherwise, shell_env changes will be overwritten and environment variables won't be available for your commands.
When executing the fab file above, you get the following output:
$ fab -H localhost my_task
[localhost] Executing task 'my_task'
[localhost] run: env | grep BAR
[localhost] out: FOO1=BAR1, FOO2=BAR2, FOO3=BAR3
[localhost] out: FOO1=BAR1
[localhost] out: FOO2=BAR2
[localhost] out: FOO3=BAR3
[localhost] out:
Done.
Disconnecting from localhost... done.

Fabric 1.5.0 (currently in Git) takes shell as local() named argument.
If you pass '/bin/bash' there it passes it to executable argument of Popen.
It won't execute your .bashrc though because .bashrc is sourced on interactive invocation of bash. You can source any file you want inside local:
local('. /usr/local/bin/virtualenvwrapper.sh && workon focus_tests && bunch local output', shell='/bin/bash')

Another way is to pass a value through command line with --set:
--set=domain=stackoverflow.com
Then, you can address to it in script with env.domain
see http://docs.fabfile.org/en/1.11/usage/fab.html#cmdoption--set

Try using decorator
from fabric.context_managers import shell_env
from functools import wraps
def set_env():
def decorator(func):
#wraps(func)
def inner(*args, **kwargs):
with shell_env(DJANGO_CONFIGURATION=env.config):
run("echo $DJANGO_CONFIGURATION")
return func(*args, **kwargs)
return inner
return decorator
#task
#set_env()
def testme():
pass