I'm trying to set the environment variable of my .bashrc using Spyder; in other words I'm looking for a python command that reads my .bashrc. Any idea?
.bashrc should automatically be loaded into the environ on login
import os
print os.environ
if you wanted to create a dictionary of values from a bash source file you could in theory do something like
output = subprocess.check_output("source /path/to/.bashrc;env")
env = dict(line.split("=") for line in output.splitlines() if "=" in line))
print env
The shell's startup file is the shell's startup file. You really want to decouple things so that Python doesn't have to understand Bash syntax, and so that settings you want to use from Python are not hidden inside a different utility's monolithic startup file.
One way to solve this is to put your environment variables in a separate file, and source that file from your .bashrc. Then when you invoke a shell from Python, that code can source the same file if it needs to.
# .bashrc
source $HOME/lib/settings.sh
# Python >=3.5+ code
import subprocess
subprocess.run(
'source $HOME/lib/settings.sh; exec the command you actually want to run',
# Need a shell; need the shell to be Bash
shell=True, executable='/bin/bash',
# basic hygiene
check=True, universal_newlines=True)
(If you need to be compatible with older Python versions, try subprocess.check_call() or even subprocess.call() if you want to give up the safeguards by the check_ variant in favor of being compatible all the way back to Python 2.4.)
Related
I need to run a python command inside a Popen. The problem is that the command NEEDS to run in python3 and I need it to be portable, which means that I can't really use the python3 alias for every situation...
I have computers where python is already the correct version, and others where the correct one is python3. I tried to insert #!/usr/bin python3 in the beginning of the file and then run as python but it didn't work.
I can't modify environment vars to change python3 to python. I would like to know if there is a way to check which one I need to use or a way to change the python3 to python ONLY inside the Popen command...
The Popen command I am trying to run is very simple and no I can't just import the file and use as a class... it needs to be ran through Popen. Also, virtualenv or similars are not an option.
subprocess.Popen(['python', 'main.py'], shell=True, universal_newlines=True)
The shebang -- the initial line showing which interpreter to use, such as #!/usr/bin/python or #!/usr/bin/python3 -- is only honored if you don't explicitly select an interpreter yourself: If you run python foo.py, the OS is invoking a specific Python interpreter and passing it foo.py as an argument (which it interprets as the name of a script it should run); whereas when you run ./foo.py, you're telling the OS itself to figure out which interpreter to use to run foo.py, which it does by looking at the shebang.
To leave it up to the operating system to select, just explicitly specify the name of your script:
subprocess.Popen(['./main.py'], universal_newlines=True)
Question
My default Python is 2.7, but I have a script that requires Python 3.4. I am trying to create a function in R that will:
Switch to Python 3.4
Run this script
Switch back to Python 2.7
Import results into R
To switch between Python versions, I use my cluster's "dotkit" system, like this:
use Python-2.7
use Python-3.4
"use" is a bash function that is imported in my .bashrc file. It sets all of my path variables (PATH, LIBRARY_PATH, LD_LIBRARY_PATH, CPATH, C_INCLUDE_PATH, etc). The problem is that when I try to call this function in R, I get the following error:
system('use Python-3.4')
sh: use: command not found
It seems like this is a problem with my PATH. I am using the correct shell:
system('echo $SHELL')
/bin/bash
My $PATH variable also looks good. However, when I create a script that essentially does the same thing:
load_py34.sh:
#!/bin/bash
source ~/.bashrc
use Python-3.4
and call this script through R, then it actually runs, but for some reason, it doesn't change my python version within R. (I have verified that this script works from the command line.)
> R
> system('python --version')
Python 2.7.1
> system('sh load_py34.sh')
Prepending: R-3.4 (ok)
> system('python --version')
Python 2.7.1
So I'm a little confused, but if anyone can help, I would really appreciate it.
Suggested fixes
When I combine them into a single command, I still have the same problem:
> system("sh load_py34.sh; python --version")
Prepending: Python-3.4 (already loaded)
Python 2.7.1
When I try calling bash directly, I still have a problem with the PATH:
> system("bash -c 'use Python-3.4; python --version'")
bash: use: command not found
Python 2.7.1
.bashrc is only loaded for interactive bash sessions.
"use" is a bash function that is imported in my .bashrc file. It sets
all of my path variables.
If set via export, the environment of the calling process will not be altered.
export [-fn] [name[=word]] ... The supplied names are marked for automatic export to the environment of subsequently executed commands. (https://man7.org/linux/man-pages/man1/bash.1.html)
Child processes do not normally have access to the parent process' environment. (This poses a problem because system() creates a sub-process.)
The source and . built-ins execute the commands in the current shell environment, hence why your script works.
Other commands (executables, non-shell-builtins) are executed by the fork-and-exec mechanism, whereby the executing shell process forks, creating a child process with an identical environment and state. This new child process is the process in which the command is executed. Changes to the environment of that process are not replicated to the parent's environment.
This means that you will not be able to rely on system('...') to modify the environment of the R process, or that of processes spawned by subsequent system() invocations.
In a single invocation to system(), you can construct a command-line that changes the environment of the spawned shell like so:
bash -c 'source ~/.bashrc; use Python-3.4; python --version'
Mind you, ~/.bashrc is not really the best place to put this functionality (might be subjective).
When you call system() it uses /bin/sh, not /bin/bash. sh doesn't read your .bashrc file when it starts up, so it does not know any of the functions you've defined there.
To use the function from your .bashrc, you must get bash to run it instead:
system("bash -c 'use Python-3.4; python --version'")
Edit: placement of closing single quote.
Running on Ubuntu, I have a python script, written by someone else, which simply runs like this:
python cool_script.py command line args
However, this script needs some environment variables set before it can run. So I created exports.sh located in the same dir as the python script, which looks something like:
export ENV1='foo'
export ENV2='bar'
export ENV3='baz'
so running:
$source exports.sh
$python cool_script.py command line args
works great. However, my case is a bit more complex. I need to run this on several environments, where the values of ENV1,ENV2,ENV3 will have to be different. I also want to do some stuff to the outputs of cool_script.py. So I wrote cool_script_wrapper.py, which looks like this:
# set environment variables
import os
exports_file = os.path.dirname(os.path.realpath(__file__)) + '/exports.sh'
os.system('source %s' % exports_file)
# run cool_script.py
os.system('python cool_script.py command line args')
# do some stuff to the output
...
I was planning to have different exports.sh scripts for each environment I need to run on, always keeping them in the same dir as the main script. Only problem is that this 'source' approach doesn't work. The environment variables are simply not available to cool_script.py.
Searching around Stack Overflow, I understand now that it's not supposed to work, but I didn't find any solution that suits my needs. I don't want to use os.environ with hard-coded values, and I don't want to parse the exports file. I guess I could make my wrapper a bash script rather than a python, but I hope there is a better solution. Any ideas?
The problem is in the source command that attempts to call a bash builtin.
This should work:
import os
os.environ['ENV1'] = "foo"
os.system("python cool_script.py command line args")
compare with this answer:
Calling the "source" command from subprocess.Popen
I have a python program that needs to run a bash command on a server. The program works when I run a bash command in my local directory like this:
import subprocess
from subprocess import call
call(['bash', 'Script_Test.sh'])
However, after SSH'ing to a server and running a similar line of code below with a path on the server to a bash script, I get the error "No such file or directory"
call['bash', path]
This doesn't make sense for a number of reasons. I triple checked that the path is correct. I went on Putty, connected to the server on there, and ran a bash command with the same path and it worked, so it can't be the path. I also ran a number of tests to make sure I was SSH'd into the correct server, and I was. I thought there was a security issue on the server with me running bash, so I tried cat instead. Nope, still unable to locate the path.
I'm not super familiar with python subprocesses, so any pointers to anything I'm missing here with "call" would be very helpful.
Making Sure Your Script is Ready for Execution
Give your script a shebang line
First things first, it is important that you include a shebang line in your script on Unix-like systems. I recommend, for your script's portability, that you use #!/usr/bin/env bash.
A Note on File Extensions:
I would recommend that you remove the .sh extension from your script. If you are using the Bourne Again Shell (bash) to execute your script, then using the .sh extension is misleading. Simply put, the Bourne Shell (sh) is different than the Bourne Again Shell (bash) - so don't use a file extension that suggests you are using a different shell than you actually are!
It's not the end of the world if you don't do change your file extension - your script will still be executed as a bash script if you have the proper bash shebang line. Still, it is good practice to either use no file extension (Script_Test -- strongly preferred) or the .bash file extension (Script_Test.bash).
Give your script the proper file permissions
For your purposes, maybe it is only important to give the current user permissions to read and execute the script. In that case, use chmod u+x Script_Test.sh. What is important here is that the correct user (u+) / group (g+) has permissions to execute the script.
Make sure that your script's path is in the $PATH environment variable
Executing your bash script in a Python script
Once you've followed these steps, your Python script should work as you've called it in your question:
import subprocess
from subprocess import call
your_call = call("Test_Script.sh")
If you would rather not move your script into the $PATH environment variable, just make sure that you refer to the script's full path (which is the current directory, ./, in your case):
import subprocess
from subprocess import call
your_call = call("./Test_Script.sh")
Lastly, if your script does not have a shebang line, you will need to specify an additional parameter in your call function:
import subprocess
from subprocess import call
your_call = call("./Test_Script.sh", shell=True)
However, I would not recommend this last approach. See the Python 2.7.12 documentation for the subprocess package...
Warning: Using shell=True can be a security hazard. See the warning under Frequently Used Arguments for details.
Please check out #zenpoy's explanation to a similar StackOverflow question if you are still having problems.
Happy coding!
I want to run a csh file from a python script,
example,
#!/usr/bin/python
import os
os.system("source path/to/file.csh")
and I want this file to run in the same shell as I am running the python script, because the file.csh script is settings some environment variables that I need.
Does anyone know how to do this in Python?
A child process cannot affect the environment of the parent process. The best you can do is to run your csh script in a separate process, get the environment variables that it defines, then set each environment variable in your python script.
Even with that, the python script won't be able to affect the shell in which you run the python script.
The common way to solve this (AFAIK) is to have your script emit shell commands to set the environment, then from the main shell you run the script and eval what you get back.
For more information you might want to check out this question: can a shell script set environment variables of the calling shell
You can kludge it this way:
#!/usr/bin/env python
# This is kludge.py
print "setenv VARNAME \"the value\""
In your case, you can have the file.sh print the setenv line.
Then from csh:
$ eval `./kludge.py`
$ echo $VARNAME
the value
This isn't clean, but it is the only way to have a child process effect the environment of its parent. This is only because the parent process is explicitly letting it happen with eval.