Develop Reference

Can I run a shell command from python script in the current shell

I have a python script which invokes two different shell scripts. The first script sets some environment variables which are required by the second script. The python code has the following structure :
import subprocess
subprocess.call(["bash", "a.sh"]) #a.sh sets env_var1
subprocess.call(["bash", "b.sh"]) #b.sh reads env_var1
Because the scripts a.sh and b.sh run in different shells, the above code does not do the needful.
I want to know that can we execute these shell scripts from python in the current shell itself

You can use this line to run commands to your shell in python os.system('command to run here')
In your case it would be something like:
import os
os.system('./a.sh')
os.system('./b.sh')

Launching script to export environment variables from Python

I'm trying to remove all passwords and secrets from applications.
To do this, I'm using environment variables. I would like to avoid putting this specific stuff into ~/.bashrc or other built in files. It would be ideal to call a specific script that exports the desired variables.
The reason for this is because the environment variables contain passwords and I'm trying to isolate them outside of all code except the environment scripts that set environment variables.
To summarize the desired flow:
Launch Python app
App spawns process that sets new environment variables
App pulls values from previously set environment variables
App uses those values in further processing
contents of testpy.py
#!/usr/bin/env python
import os
import subprocess
env_file='/tmp/env_file'
db_pass_var='DBPASS'
#execute bash script that sets the env vars
subprocess.call([".", env_file], shell=True)
#try to get the variable set and print it
print os.getenv(db_pass, 'fail')
contents of /tmp/env_file
#!/usr/bin/env bash
export DBUSER="myuser"
export DBPASS="mypass"
I've tried various methods like subprocess.Popen commands I found on other stackoverflow threads but nothing seems to work.
Alternatives that I want to avoid:
Running the env_file script first, then launching the python code. The goal is to run it all from 1 application.
Putting the passwords into a config file. The goal is environment variables. Config files I would like to reserve to pointing to what variable to call, rather than setting the env vars themselves. This is because config files are checked into source control. The only thing I want to leave out of source control is the actual export environment variables themselves which contain all sensitive environment info.

There's an easier way. Just provide a wrapper script that sets the environment variables and then calls your main script.
It's convenient to use the envdir utility for this. If you can't use that for some reason, then set env vars directly using os.environ and spawn off your main script as a child process.

Set shell environment variable via python script

I have some instrument which requires environment variable which I want to set automatically from python code. So I tried several ways to make it happen, but none of them were successful.
Here are some examples:
I insert following code in my python script
import os
os.system("export ENV_VAR=/some_path")
I created bash script(env.sh) and run it from python:
#!/bin/bash
export ENV_VAR=some_path
#call it from python
os.system("source env.sh")
I also tried os.putenv() and os.environ*["ENV_VAR"] = "some_path"
Is it possible to set(export) environment variable using python, i.e
without directly exporting it to shell?

Setting an environment variable sets it only for the current process and any child processes it launches. So using os.system will set it only for the shell that is running to execute the command you provided. When that command finishes, the shell goes away, and so does the environment variable. Setting it using os.putenv or os.environ has a similar effect; the environment variables are set for the Python process and any children of it.
I assume you are trying to have those variables set for the shell that you launch the script from, or globally. That can't work because the shell (or other process) is not a child of the Python script in which you are setting the variable.
You'll have better luck setting the variables in a shell script. If you then source that script (so that it runs in the current instance of the shell, rather than in a subshell) then they will remain set after the script ends.

As long as you start the "instrument" (a script I suppose) from the very same process it should work:
In [1]: os.putenv("VARIABLE", "123")
In [2]: os.system("echo $VARIABLE")
123
You can't change an environment variable of a different process or a parent process.

A shell function may do this. You need to print your export statement and eval that.
set_shell_env() {
output=$(python print_export_env.py $*)
eval $output
}

Depending on how you execute your instrument, you might be able to change environment specifically for the child process without affecting the parent. See documentation for os.spawn*e or subprocess.Popen which accept separate argument denoting child environment. For example, Replacing the os.spawn family in subprocess module documentation which provides both usages:
Environment example:
os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
==>
Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})

Trying to get an enviroment variable in python, not working

Say I do this in the terminal
TEST="abc"
A python script run after this (same session, variable is definitely still there) raises a KeyError as the key TEST doesn't exist. How do I access this environment variable?
import os
print os.environ["TEST"]

# bash
export TEST=abc
# sh
TEST=abc
export TEST
Make sure to export the variable. By default environment variables are not inherited by child processes. Marking them as exported tells the shell to pass them to its children.

In the terminal, do
export TEST="abc"

problem with python script

I want to run a csh file from a python script,
example,
#!/usr/bin/python
import os
os.system("source path/to/file.csh")
and I want this file to run in the same shell as I am running the python script, because the file.csh script is settings some environment variables that I need.
Does anyone know how to do this in Python?

A child process cannot affect the environment of the parent process. The best you can do is to run your csh script in a separate process, get the environment variables that it defines, then set each environment variable in your python script.
Even with that, the python script won't be able to affect the shell in which you run the python script.
The common way to solve this (AFAIK) is to have your script emit shell commands to set the environment, then from the main shell you run the script and eval what you get back.
For more information you might want to check out this question: can a shell script set environment variables of the calling shell

You can kludge it this way:
#!/usr/bin/env python
# This is kludge.py
print "setenv VARNAME \"the value\""
In your case, you can have the file.sh print the setenv line.
Then from csh:
$ eval `./kludge.py`
$ echo $VARNAME
the value
This isn't clean, but it is the only way to have a child process effect the environment of its parent. This is only because the parent process is explicitly letting it happen with eval.