Develop Reference

Activating conda environment with cloud-init

We are trying to run batch scripts on load on a AWS EC2 instance using userdata (which I understand is based off of cloud-init). Since the code runs in a conda environment, we are trying to activate it prior to running the Python/Pandas code. We noticed that the PATH variable isn't getting set correctly. (even though it was set correctly prior to making the image, and is set correctly for all users after SSH'ing into instance)
We've tried modifiying the path in the shell script, but nothing is sticking. We ran the following code:
#!/bin/bash
export=/opt/conda/bin:$PATH
which python
which conda
conda activate etl
We checked $PATH before and after running export=/opt/conda/bin (no change). which python returns the wrong python, and which conda returns not found.
$PATH before and after is:
/sbin:/usr/sbin:/bin:/usr/bin

After hours of work, the two critical lines needed (regardless of your usage of the Miniconda AMI) is:
#!/bin/bash
echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc #or the path to your conda.sh
source ~/.bashrc
conda activate <full path to environment>
for some reason, cloud-init ignores all other attempts to modify path

Activating conda environment in bash script that runs on startup

So I have a python script that generates an animation - and it requires libraries that I have in a conda environment. I need to run this script as soon as my computer turns on, so I've written a short bash script that I added to "startup applications". This bash script runs on startup, and reads like this:
#!/bin/bash
conda activate myenv
cd ~/scripts
python generate.py
When I run this in terminal myself, it's fine, but whenever I turn on the computer, the python part of the script doesn't execute, and when I check errors i find:
conda: command not found
and then i also see the python script failed to run because it's missing libraries (from the conda environment not activating)
I have tried adding lines to the bash script replacing "conda activate" with "source activate", I have tried adding echo ". /home/<user>/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc to the bash script, replacing "conda" with /home/barrat/anaconda3/bin/conda, and even adding whoami to the bash script that runs at startup to make sure that i haven't magically become root by chance... none of this has worked. I would really appreciate any help. it's 3 AM and i'm a bit desperate.

You might have solved the issue yet, but for future viewers, this worked for me:
if [ -f "/path/to/anaconda3/etc/profile.d/conda.sh" ]; then
. "/path/to/anaconda3/etc/profile.d/conda.sh"
CONDA_CHANGEPS1=false conda activate myenv
fi
Add this instead of conda activate myenv.

Well as you are trying to activate an environment to start your scripts, it may also be possible for you to make a startup script itself to do the desired task by using subprocess module from python.
Try making a demo.py script like :
import os
import system
import subprocess
import x
subprocess.run(["command name", "value"]) #for all scripts you want to execute
and then you can put this python script to run at startup.
You can start quite a few amount of operations without noticable speed changes to your system and always can monitor it easily by starting the processes one after the other using time.sleep() in between two calls.

How to grant access to virtual environment to other users on CentOS server

I used this tutorial (rec by GoDaddy because I have a GoDaddy VPS)
https://www.godaddy.com/garage/how-to-install-and-configure-python-on-a-hosted-server/
to install a secondary version of Python (3.6.6) and create a Virtual Environment to run modules with the secondary version of Python. I want to be able to run scripts in the virtual environment but do not want to always use the admin account to do this. I have one other account set up that does not have admin privileges. Is there a way to extend authority to more than the admin account that created it? Do I need to do a virtual environment on every user account that is going to have access to run python?
It says to add a line to the bash_profile script. The script already had this:
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
This makes sense in that it seems to be defining the location of underlying Python (i.e., 2.6.6). I have modules in a venv that require 3.4+ so I created the virtual environment. I want to be able to call the script from Chrome and run it for some scripts and run it from cronjob for other scripts and the scripts need to always point to the virtualenv to use that installed interpreter. I have no idea how to do this correctly. Here's my system information and the results of some SSH printouts:
$ cd $PATH
-bash: cd: /home/flohosti/.local/bin:/home/flohosti/perl5/bin:/usr/local/cpanel/3rdparty/lib/path-bin:/usr/local/cpanel/3rdparty/lib/path-bin:/usr/local/cpanel/3rdparty/lib/path-bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/cpanel/composer/bin: No such file or directory
Before going into the VirtualEnv I run which python and get /usr/bin/python which I expected.

There are multiple ways to do that.
You can simply give read access to the directory where virtual environment is installed and then export its path in /etc/profile so that its automatically added to every users path.
Or you can give access to python binary inside the virtual env folder.
That being said this isn't the right way to go about it, most of the times you would want to have virtual environment inside your projet folder so that it is automatically available the project and who so ever has access to that project.
Edit 1
So the answer to your comment
Yes absolutely you can use the same virtual env to execute script in cron but you have to give absolute path to virtaul environment and your script in cron, like /home/flohosti/projectOne/env/bin/python <script path> or you can just set the PATH in cron and then you can use relative path as well.

Is there a single line way to run a command in a Python venv?

I have a command that only runs correctly inside a Python virtual environment I've configured (as intended). I know that I can run the command as
$ cmd args
once I've activated the venv. But (due to the constraints of the tool I'm using) I need to activate run (and deactivate?) in one line: something equivalent to running
$ activate_somehow cmd args
outside the command line.
Is there a way to do this?

You can generally run something in a virtual environment simply by using a fully qualified path to the script. For example, if I have:
virtualenv .venv
Then I can install something into that virtual environment without activating it by running:
.venv/bin/pip install foo
This should be true for anything installed using standard Python mechanisms.

After looking into the generated bin/activate script, it seems like the only thing relevant to python is the VIRTUAL_ENV variable, so this should be enough to get going:
$ env VIRTUAL_ENV=path/to/venv python ...
Note that the python executable in the bin directory of target environment is just a symlink to globally installed interpreter, which does nothing other that setting process executable path. Assuming the program does not make use of it, utilizing the main binary itself seems harmless. In case you have installed a package which in turn installs some executables, just specify the absolute path:
$ env VIRTUAL_ENV=path/to/venv path/to/venv/bin/executable

You can create a simple wrapper script which runs activate, executes your command, and then deactivates simply by exiting the script in which your environment was activated.
#!/bin/sh
. ${venv-./env}/bin/activate
"$#"
This lets you set the environment variable venv to the path of the environment you want to use, or else uses ./env if it is unset. Perhaps a better design would be to pass the env as the first parameter:
#!/bin/sh
. "$1"/bin/activate
shift
"$#"
Either way, save this somewhere in your PATH ($HOME/bin is a common choice for your private scripts) and give it executable permission.

I found venv-run which should do what you ask:
pip install venv-run
venv-run cmd args

Larsk's answer is probably cleaner, but this is another possible way.
Assuming you use UNIX and your user is user and you have a virtual environment in home (any) directory, ie /home/user/venv, you can make a script like:
#!/bin/sh
export VIRTUAL_ENV=/home/user/venv
export PATH=/home/user/venv/bin:$PATH
python3 "$#"
We can make this script executable (eg call it venv-python3 and do chmod +x venv-python3) and call it as such, or put it some place discoverable in PATH - let's say alongside python. Assuming you have sudo rights:
sudo cp venv-python3 /usr/bin/venv-python3
Then we can call that instead of the python callable. Since the variables are set within the script, explicit call on deactivate is not necessary at exit.
Example:
user#machine ~ % venv-python3 --help
This works for at least for virtualenv version 20.0.17 but if adopted, you should be keeping an eye on what variables bin/activate sets, if this ever changes.

Yes, you can execute the python file using a virtual environment in a single line of command on windows.
venv\Scripts\activate&&python fall_detector.py

I installed pgadmin4 in my home directory in a virtual environment called "pgadmin4".
I use fish shell and it runs perfectly fine with:
~/pgadmin4/bin/python3 ~/pgadmin4/lib/python3.10/site-packages/pgadmin4/pgAdmin4.py
Just in case this helps somebody.

Mercurial update hook not activating Python virtual environment

I have a bash script that I'm trying to execute anytime an hg update occurs. The goal of this bash script is to switch to the correct virtualenv. For the sake of simplicity, this script is called .test - shown as follows:
#!/bin/bash
echo 'testing hg update hook'
source ~/.virtualenvs/myvirtualenv/bin/activate
Whenever I call this script from my shell using source .test everythying works properly; I can see the results of echo and my shell changes to reflect the activated virtualenv
However,when I do an hg update, the virtualenv is not being activated. The script is firing as I can see the echo result; however, my shell is not updated to reflect the activated virtualenv. Below is the hook setup in my .hg/hgrc file is below. Any ideas why my virtualenv isn't being activated in this hook?
[hooks]
# Update to the correct virtualenv when switching branches (hg update branchname)
update = source .test
UPDATE 1: Per this answer, I don't believe the hg update hook is firing in my current shell; which is why the virtualenv activates when I run the script manually but fails from the hook

Your problem is that when you invoke a shell script, any changes to the environment variables do not get exported to the calling shell (hence why you need to call source activate from the surrounding shell).
The good news is that you don't strictly need to call activate in order to access a virtual environment. What activate will do is:
Add the virtualenv's bin directory to $PATH.
Set the VIRTUAL_ENV environment variable.
Modify your prompt.
None of this is necessary in order to use the virtualenv, and you can execute the python binary in the virtualenv without ever using the script; the prompt is likely not relevant for your use case, you can add the directory (or just the python executable) to your path by symlinking it, and you need the VIRTUAL_ENV environment variable only for software that for some reason needs to be aware of the virtualenv it's running in. If necessary, you can figure it out from sys.executable. For example:
import sys, os
def find_venv():
python = sys.executable
for i in xrange(10):
if not os.path.islink(python):
break
python = os.path.realpath(python)
return os.path.dirname(os.path.dirname(python))
if not os.environ.has_key("VIRTUAL_ENV"):
os.environ["VIRTUAL_ENV"] = find_venv()