I am trying to create a shell script that executes some python scripts in order.
I have never done something like this and i don´t know if what i am doing is correct.
Before executing the last file i need to activate an enviroment so it has the necesary imports to so it functions correctly.
This is the script i have created (execute.sh):
#!/bin/sh
python path/file_1.py
python path/file_2.py
python path/file_3.py
# activate a virtual enviroment
source /path/env/bin/activate
python path/file_4.py
After this what sould I do?
Just write this line in the terminal?
./execute.sh
That should work. Usually I always do this when running a bash script with Python scripts inside.
#!/bin/bash
. /home/user/.bashrc # Load basic env as we're coming from cron
# Cd into our working directory in case we're not into it already
cd "$(dirname "$0")";
echo "----------------"
echo "Starting processing `date`"
echo "----------------"
conda activate models # Activate my conda environment
# Execute Python script
python script.py
if [[ $? = 0 ]]; then
echo "Succesfull execution!"
else
echo "Something wrong"
exit 1
fi
This is in execute.run, which you then make executable chmod +x executable.run, and run. This will work wherever you are because it will always try to cd into the directory where the script resides, so you can also call it from crontab.
I have few conda environments that I use in different projects, say:
ml37 (for machine learning)
etl37 (for data pipelines)
I have local projects organized in their own directories:
apps/some_app
apps/other_app
...
Each time I cd to a specific project, I already know which env I would like to use. So I end up doing conda activate [some env] each time I change directories. I feel like there must be a better way.
What would be a clean way to automatize this?
Or is my use of conda environments wrong?
I made a script similar to the one of Corey Chafer, but this one extends the cd command.
cd() { builtin cd "$#" &&
if [ -f $PWD/.conda_config ]; then
export CONDACONFIGDIR=$PWD
conda activate $(cat .conda_config)
elif [ "$CONDACONFIGDIR" ]; then
if [[ $PWD != *"$CONDACONFIGDIR"* ]]; then
export CONDACONFIGDIR=""
conda deactivate
fi
fi }
Put this few lines of code at the bottom of your shell profile and then create a .conda_config file inside the directory you want to activate the env for.
The .conda_config file must contain only the env name.
In this way, every time you cd into a directory that has a .conda_config file, the script will activate the env, and every time you cd out it will deactivate.
I created a repo for reference Conda-autoactivate-env
EDIT:
There was a bug in the elif condition.
Basically [-n $CONDACONFIGDIR] returns always True and its logic is backwards actually.
The fix is:
quote the variable (or use double squared brackets) and remove -n
[ "$CONDACONFIGDIR" ] OR [[ $CONDACONFIGDIR ]].
The above code is already up-to-date!
I am trying to write a simple script that will help me activate (source) the virtualenv and set some environment variables at the same time. Below is the current version that does not yet have any environment variables.
#!/bin/bash
if [ -z "$BASH_VERSION" ]
then
exec bash "$0" "$#"
fi
# Your script here
script_dir=`dirname $0`
cd $script_dir
/bin/bash -c ". ./django-env/bin/activate; exec /bin/bash -i"
The trouble with this script is two-fold.
When I run it - seemingly successfully as it changes the command line prefix to (django-env) - it is missing the My-Computer-Name: in front of it. Obviously, it is an indication of something as I typically have (django-env) My-Computer-Name: as the prefix.
It does not activate the virtualenv properly. Namely, when I check which python, I am notified that the virtualenv Python is used. On the other hand, when I check for which pip or which python3, the global system's Python is used.
What can I do to fix these things and have the environment be activated?
I suspect the problem with exec /bin/bash -i — the executed bash could run .bash_profile and .bashrc that change the current environment.
Instead of a shell script that executes shells upon shells you better create an alias or a shell function:
django_activate() {
cd $1
. ./django-env/bin/activate
}
Put it in .bashrc so it will be available in all shells and run as django_activate $venv_dir; for example django_activate ~/projects/work.
The following code does what I intended it to do. I run it with source script.sh
#!/bin/bash
if [ -z "$BASH_VERSION" ]
then
exec bash "$0" "$#"
fi
# Your script here
script_dir=`dirname $0`
cd $script_dir
/bin/bash -c ". ./django-env/bin/activate"
I'm executing with pycharm the following:
print(os.environ["PATH"]) # returns '/usr/bin:/bin:/usr/sbin:/sbin'
But when I execute echo $PATH in a shell this is returned:
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/local/bin:/opt/local/sbin
I tried to edit it in Preferences > Console > Python Console > Environment Variables, setting
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/local/bin:/opt/local/sbin
but this is not working
any idea?
On Ubuntu, using zsh, I stumbled upon the same problem.
The hack I use in order to have the same environment variables in PyCharm and my shell is to launch PyCharm from my terminal instead of using the icon. It looks like this way that the PyCharm shell inherits from the main shell it's been launched from.
I hope it can solve other people's problem as I wasn't able to replicate #Steve Tarver's solution on Linux (.../terminal/.zshrc was read only on /snap/, even when using sudo).
#fj123x, I'm going to guess from your post that you are
on a mac
using a shell other than bash (perhaps zsh)
If true, the problem is that the JetBrains jediterm terminal emulator is not executing all shell startup files in the correct order.
If you are using zsh, you can fix that root problem by editing the terminal plugin's .zshrc. PyCharm is in your Applications folder, open /Applications/PyCharm.app/Contents/plugins/terminal/.zshrc and replace the contents with:
#!/bin/zsh
# starver mod
# Jetbrains uses jediterm as a java terminal emulator for all terminal uses.
# There are some apparent limits on use:
# - must use old-style shebang - not the #!/usr/bin/env zsh
# - must implement the startup file loading here
#
# Note: original contents are in lib/terminal.jar
# mappings for Ctrl-left-arrow and Ctrl-right-arrow for word moving
bindkey '^[^[[C' forward-word
bindkey '^[^[[D' backward-word
ZDOTDIR=$_OLD_ZDOTDIR
if [ -n "$JEDITERM_USER_RCFILE" ]
then
source "$JEDITERM_USER_RCFILE"
unset JEDITERM_USER_RCFILE
fi
if [ -n "$ZDOTDIR" ]
then
DOTDIR=$ZDOTDIR
else
DOTDIR=$HOME
fi
if [ -f "/etc/zshenv" ]; then
source "/etc/zshenv"
fi
if [ -f "$DOTDIR/.zshenv" ]; then
source "$DOTDIR/.zshenv"
fi
if [ -n $LOGIN_SHELL ]; then
if [ -f "/etc/zprofile" ]; then
source "/etc/zprofile"
fi
if [ -f "$DOTDIR/.zprofile" ]; then
source "$DOTDIR/.zprofile"
fi
fi
if [ -f "/etc/zshrc" ]; then
source "/etc/zshrc"
fi
if [ -f "$DOTDIR/.zshrc" ]; then
source "$DOTDIR/.zshrc"
fi
if [ -n $LOGIN_SHELL ]; then
if [ -f "/etc/zlogin" ]; then
source "/etc/zlogin"
fi
if [ -f "$DOTDIR/.zlogin" ]; then
source "$DOTDIR/.zlogin"
fi
fi
if [ -n "$JEDITERM_SOURCE" ]
then
source $(echo $JEDITERM_SOURCE)
unset JEDITERM_SOURCE
fi
If you are interested in all the gory details, or you want to see how I solved this problem so you can develop a solution for another shell, see this answer: https://stackoverflow.com/a/51006003/1089228
I work on the command line in bash and my environment, including $PATH, is set in .bash_profile. The default terminal in PyCharm is tcsh.
I changed it to bash by going
File ... Default Settings ... Tools ... Terminal ... Shell Path
and then restaring. The embedded terminal worked as expected.
How do you create a Bash script to activate a Python virtualenv?
I have a directory structure like:
.env
bin
activate
...other virtualenv files...
src
shell.sh
...my code...
I can activate my virtualenv by:
user#localhost:src$ . ../.env/bin/activate
(.env)user#localhost:src$
However, doing the same from a Bash script does nothing:
user#localhost:src$ cat shell.sh
#!/bin/bash
. ../.env/bin/activate
user#localhost:src$ ./shell.sh
user#localhost:src$
What am I doing wrong?
When you source, you're loading the activate script into your active shell.
When you do it in a script, you load it into that shell which exits when your script finishes and you're back to your original, unactivated shell.
Your best option would be to do it in a function
activate () {
. ../.env/bin/activate
}
or an alias
alias activate=". ../.env/bin/activate"
You should call the bash script using source.
Here is an example:
#!/bin/bash
# Let's call this script venv.sh
source "<absolute_path_recommended_here>/.env/bin/activate"
On your shell just call it like that:
> source venv.sh
Or as #outmind suggested: (Note that this does not work with zsh)
> . venv.sh
There you go, the shell indication will be placed on your prompt.
Although it doesn't add the "(.env)" prefix to the shell prompt, I found this script works as expected.
#!/bin/bash
script_dir=`dirname $0`
cd $script_dir
/bin/bash -c ". ../.env/bin/activate; exec /bin/bash -i"
e.g.
user#localhost:~/src$ which pip
/usr/local/bin/pip
user#localhost:~/src$ which python
/usr/bin/python
user#localhost:~/src$ ./shell
user#localhost:~/src$ which pip
~/.env/bin/pip
user#localhost:~/src$ which python
~/.env/bin/python
user#localhost:~/src$ exit
exit
Sourcing runs shell commands in your current shell. When you source inside of a script like you are doing above, you are affecting the environment for that script, but when the script exits, the environment changes are undone, as they've effectively gone out of scope.
If your intent is to run shell commands in the virtualenv, you can do that in your script after sourcing the activate script. If your intent is to interact with a shell inside the virtualenv, then you can spawn a sub-shell inside your script which would inherit the environment.
Here is the script that I use often. Run it as $ source script_name
#!/bin/bash -x
PWD=`pwd`
/usr/local/bin/virtualenv --python=python3 venv
echo $PWD
activate () {
. $PWD/venv/bin/activate
}
activate
You can also do this using a subshell to better contain your usage - here's a practical example:
#!/bin/bash
commandA --args
# Run commandB in a subshell and collect its output in $VAR
# NOTE
# - PATH is only modified as an example
# - output beyond a single value may not be captured without quoting
# - it is important to discard (or separate) virtualenv activation stdout
# if the stdout of commandB is to be captured
#
VAR=$(
PATH="/opt/bin/foo:$PATH"
. /path/to/activate > /dev/null # activate virtualenv
commandB # tool from /opt/bin/ which requires virtualenv
)
# Use the output from commandB later
commandC "$VAR"
This style is especially helpful when
a different version of commandA or commandC exists under /opt/bin
commandB exists in the system PATH or is very common
these commands fail under the virtualenv
one needs a variety of different virtualenvs
What does sourcing the bash script for?
If you intend to switch between multiple virtualenvs or enter one virtualenv quickly, have you tried virtualenvwrapper? It provides a lot of utils like workon venv, mkvirtualenv venv and so on.
If you just run a python script in certain virtualenv, use /path/to/venv/bin/python script.py to run it.
As others already stated, what you are doing wrong is not sourcing the script you created. When you run the script just like you showed, it creates a new shell which activates the virtual environment and then exits, so there are no changes to your original shell from which you ran the script.
You need to source the script, which will make it run in your current shell.
You can do that by calling source shell.sh or . shell.sh
To make sure the script is sourced instead of executed normally, its nice to have some checks in place in the script to remind you, for example the script I use is this:
#!/bin/bash
if [[ "$0" = "$BASH_SOURCE" ]]; then
echo "Needs to be run using source: . activate_venv.sh"
else
VENVPATH="venv/bin/activate"
if [[ $# -eq 1 ]]; then
if [ -d $1 ]; then
VENVPATH="$1/bin/activate"
else
echo "Virtual environment $1 not found"
return
fi
elif [ -d "venv" ]; then
VENVPATH="venv/bin/activate"
elif [-d "env"]; then
VENVPATH="env/bin/activate"
fi
echo "Activating virtual environment $VENVPATH"
source "$VENVPATH"
fi
It's not bulletproof but it's easy to understand and does its job.
You should use multiple commands in one line. for example:
os.system(". Projects/virenv/bin/activate && python Projects/virenv/django-project/manage.py runserver")
when you activate your virtual environment in one line, I think it forgets for other command lines and you can prevent this by using multiple commands in one line.
It worked for me :)
When I was learning venv I created a script to remind me how to activate it.
#!/bin/sh
# init_venv.sh
if [ -d "./bin" ];then
echo "[info] Ctrl+d to deactivate"
bash -c ". bin/activate; exec /usr/bin/env bash --rcfile <(echo 'PS1=\"(venv)\${PS1}\"') -i"
fi
This has the advantage that it changes the prompt.
As stated in other answers, when you run a script, it creates a sub-shell.
When the script exits, all modifications to that shell are lost.
What we need is actually to run a new shell where the virtual environment is active, and not exit from it.
Be aware, this is a new shell, not the one in use before you run your script.
What this mean is, if you type exit in it, it will exit from the subshell, and return to the previous one (the one where you ran the script), it won't close your xterm or whatever, as you may have expected.
The trouble is, when we exec bash, it reads its rc files (/etc/bash.bashrc, ~/.bashrc), which will change the shell environment. The solution is to provide bash with a way to setup the shell as usual, while additionnally activate the virtual environment. To do this, we create a temporary file, recreating the original bash behavior, and adding a few things we need to enable our venv. We then ask bash to use it instead of its usual rc files.
A beneficial side-effect of having a new shell "dedicated" to our venv, is that to deactivate the virtual environment, the only thing needed is to exit the shell.
I use this in the script exposed below to provide a 'deactivate' option, which acts by sending a signal to the new shell (kill -SIGUSR1), this signal is intercepted (trap ...) and provoke the exit from the shell.
Note: i use SIGUSR1 as to not interfere with whatever could be set in the "normal" behavior.
The script i use:
#!/bin/bash
PYTHON=python3
myname=$(basename "$0")
mydir=$(cd $(dirname "$0") && pwd)
venv_dir="${mydir}/.venv/dev"
usage() {
printf "Usage: %s (activate|deactivate)\n" "$myname"
}
[ $# -eq 1 ] || { usage >&2; exit 1; }
in_venv() {
[ -n "$VIRTUAL_ENV" -a "$VIRTUAL_ENV" = "$venv_dir" -a -n "$VIRTUAL_ENV_SHELL_PID" ]
}
case $1 in
activate)
# check if already active
in_venv && {
printf "Virtual environment already active\n"
exit 0
}
# check if created
[ -e "$venv_dir" ] || {
$PYTHON -m venv --clear --prompt "venv: dev" "$venv_dir" || {
printf "Failed to initialize venv\n" >&2
exit 1
}
}
# activate
tmp_file=$(mktemp)
cat <<EOF >"$tmp_file"
# original bash behavior
if [ -f /etc/bash.bashrc ]; then
source /etc/bash.bashrc
fi
if [ -f ~/.bashrc ]; then
source ~/.bashrc
fi
# activating venv
source "${venv_dir}/bin/activate"
# remove deactivate function:
# we don't want to call it by mistake
# and forget we have an additional shell running
unset -f deactivate
# exit venv shell
venv_deactivate() {
printf "Exitting virtual env shell.\n" >&2
exit 0
}
trap "venv_deactivate" SIGUSR1
VIRTUAL_ENV_SHELL_PID=$$
export VIRTUAL_ENV_SHELL_PID
# remove ourself, don't let temporary files laying around
rm -f "${tmp_file}"
EOF
exec "/bin/bash" --rcfile "$tmp_file" -i || {
printf "Failed to execute virtual environment shell\n" >&2
exit 1
}
;;
deactivate)
# check if active
in_venv || {
printf "Virtual environment not found\n" >&2
exit 1
}
# exit venv shell
kill -SIGUSR1 $VIRTUAL_ENV_SHELL_PID || {
printf "Failed to kill virtual environment shell\n" >&2
exit 1
}
exit 0
;;
*)
usage >&2
exit 1
;;
esac
I simply added this into my .bashrc-personal config file.
function sv () {
if [ -d "venv" ]; then
source "venv/bin/activate"
else
if [ -d ".venv" ]; then
source ".venv/bin/activate"
else
echo "Error: No virtual environment detected!"
fi
fi
}