We are trying to run batch scripts on load on a AWS EC2 instance using userdata (which I understand is based off of cloud-init). Since the code runs in a conda environment, we are trying to activate it prior to running the Python/Pandas code. We noticed that the PATH variable isn't getting set correctly. (even though it was set correctly prior to making the image, and is set correctly for all users after SSH'ing into instance)
We've tried modifiying the path in the shell script, but nothing is sticking. We ran the following code:
#!/bin/bash
export=/opt/conda/bin:$PATH
which python
which conda
conda activate etl
We checked $PATH before and after running export=/opt/conda/bin (no change). which python returns the wrong python, and which conda returns not found.
$PATH before and after is:
/sbin:/usr/sbin:/bin:/usr/bin
After hours of work, the two critical lines needed (regardless of your usage of the Miniconda AMI) is:
#!/bin/bash
echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc #or the path to your conda.sh
source ~/.bashrc
conda activate <full path to environment>
for some reason, cloud-init ignores all other attempts to modify path
Related
I use zsh but not equipted with oh-my-zsh, since I cloned many git repos on my machine and oh-my-zsh's default theme or 'josh' theme makes it very slow, for cd and ls commands, etc.
Thus I delete oh-my-zsh and only use zsh's default recommended config, except adding some alias and path variables.
Now comes the question: I manage my python environment by conda, and no current enviroment name displayed in my prompt.
I have executed the command:
conda init zsh
And re-open my terminal (default shell is zsh), still not display "(base)" or "(py36)" stuffs.
How can I tell my terminal to show that stuff? The git branch does not matters for my work, but the current python version really matters, since DeepLearning related tools use many different python versions.
For those users with oh-my-zsh
I add anaconda element in ~/.zshrc config file
POWERLEVEL9K_LEFT_PROMPT_ELEMENTS=(anaconda ...ENVS)
and the conda env shows up
If you're not using oh-my-zsh and have not copy-pasted the relevant sections from there, then the correct answer is that when you do conda init zsh, the environment variable CONDA_PROMPT_MODIFIER is introduced. You want to add that to your PROMPT. You should do so as part of the precmd function, so that the an updated value of the environment variable is shown every time a prompt is printed.
This was caused due to this line in default zsh's config:
prompt adam1
Once delete that line, re-open terminal, conda environment are shown.
You can try,
# init your shell first
conda init zsh
source ~/.zshrc # refresh your shell
source activate base # (optional, on linux) activate base env
I'm trying to set up a job file to run on a linux cluster using
$ qsub network.job
I have loaded the module needed to execute the script, however a also need to activate a virtual environment that comes with the module.
#$ -S /bin/bash
#$ -cwd
#$ -j y
#$ -o output/
echo "Running job script"
module load python/python3
./network.py
echo "Finished job script"
Including
$source activate machinelearning
does not activate the virtualenv
What do I need to add to the file to activate the virtualenv machine learning that comes when loading the module python/python3?
source /PATH/WHERE/YOUR/VIRTUAL/ENVIRONMENT/EXISTS/bin/activate
for example in the subdirectory venv of your homedir
source ~/venv/machinelearning/bin/activate
Judging by your comment response, the source command may not be the problem. The typical approach in python virtual environments is to call source on the activate script inside the virtual environment. For more information on source, see this superuser post.
In this case, since the command source activate machinelearning works for you in an interactive session, it's likely that there's a script called activate in your home directory (or whichever directory you were working from) that takes machinelearning as an argument, and activates the appropriate virtual environment.
Here's my best stab at solving your immediate problem:
Since you are able to activate the virtual environment in an interactive session, log in to an interactive session and activate the virtual environment. Once you have done so, type which python3 in your terminal, and it should give you the full path to the virtual environment you're working in. Take that full path and replace the last python3 with activate, and that is the script you should source. For example:
$>which python3
/home/cmf05/environments/machinelearning/bin/python3
Then place this in your script between module load python/python3 and ./network.py:
source /home/cmf05/environments/machinelearning/bin/activate
As an aside: module load isn't something that's built in to any systems I've ever worked on, but it looks like you're submitting a job to a cluster that's managed by slurm or similar. It seems like this is loading all of the specific files you need for your compute job. I would recommend tagging the question with your specific cluster computing environment and putting that in the title for more focused help, in case this doesn't solve your issue.
So if I load a conda environment and want to use conda to install a package (or whatever else), I have to specify conda.exe.
What I expect to work:
(randomenv) λ conda --version
The system cannot find the path specified.
What actually works:
(randomenv) λ conda.exe --version
conda 4.5.4
Further complicating things, the following:
(randomenv) λ where conda
C:\Users\[USERNAME]\AppData\Local\conda\conda\envs\randomenv\Scripts\conda
C:\Users\[USERNAME]\AppData\Local\conda\conda\envs\randomenv\Scripts\conda.bat
C:\ProgramData\Anaconda3\Scripts\conda.exe
Is the first entry supposed to be C:\Users\[USERNAME]\AppData\Local\conda\conda\envs\randomenv\Scripts\conda.exe and the environment got setup wrong or something?
Edit:
We're now in to usage breaking issues. I can't use deactivate, because deactivate doesn't have a .exe extension.
Per usual, where deactivate turns up:
C:\Users\[USERNAME]\AppData\Local\conda\conda\envs\randomenv\Scripts\deactivate
C:\Users\[USERNAME]\AppData\Local\conda\conda\envs\randomenv\Scripts\deactivate.bat
C:\ProgramData\Anaconda3\Scripts\deactivate
C:\ProgramData\Anaconda3\Scripts\deactivate.bat
Even more hilariously frustrating:
(randomenv) λ C:\Users\[USERNAME]\AppData\Local\conda\conda\endevs\randomenv\Scripts\deactivate
The system cannot find the path specified.
and then it gets even better:
(randomenv) λ cd C:\Users\[USERNAME]\AppData\Local\conda\conda\envs\randomenv\Scripts\
C:\Users\[USERNAME]\AppData\Local\conda\conda\envs\randomenv\Scripts
(randomenv) λ ls
2to3.exe* conda* easy_install.exe* epylint-script.py iptest3.exe* ipython-script.py pip-script.py pygmentize-script.py pyreverse.bat symilar.exe*
2to3-script.py conda.bat easy_install-script.py idle.exe* ipython.exe* isort.exe* pydoc.exe* pylint.bat pyreverse.exe* symilar-script.py
activate* deactivate* epylint.bat idle-script.py ipython3.exe* isort-script.py pydoc-script.py pylint.exe* pyreverse-script.py wheel.exe*
activate.bat deactivate.bat epylint.exe* iptest.exe* ipython3-script.py pip.exe* pygmentize.exe* pylint-script.py symilar.bat wheel-script.py
C:\Users\[USERNAME]\AppData\Local\conda\conda\envs\randomenv\Scripts
(randomenv) λ deactivate
The system cannot find the path specified.
How can't it find a file that's in the current directory?
Edit2:
Found the following issue on GitHub which says that exiting an environment requires that you activate root. That isn't working either, because activate can't be found just like all the other functions listed above.
This just happened to me. Here is what I found out.
Within the scripts folder of my environment were three scripts:
activate.bat
conda.bat
deactivate.bat
along with three more without the .bat extension that were bash scripts. In each case, all they did was a delegate to the globally installed copy of the same tool using an absolute path name.
When given a command, such as "conda" at the prompt, Windows will search for a .com, .exe or .bat file to execute, taking the first one it finds. Because activating an environment adds it's locations to the head of the path, the first instance windows will find is the .bat file within the environment, which will be executed.
In my case, the .bat files had the wrong absolute path, and were trying to delegate to a global copy that did not exist, hence the error message. Even if you were in the environment script directory, you would still get the message because it's not the .bat file that the message refers to, but the program the .bat file delegates to.
When specifying conda.exe, you are specifically asking for the .exe file, which means the .bat file will not be found and run, and therefore the global copy is found, which of course works.
I fixed it by changing the path inside each of the .bat files to the proper path to the tool, according to the output of where x, which fixed the problem.
I had the same problem. Interestingly, when I set up a fresh environment the activate / deactivate commands work fine.
I compared the script folder for this environment with the broken environment. The broken environment had activate / deactive files in the scripts folder just like yours, but the working environment did not.
By deleting the extraneous scripts (deactivate, conda, activate, .pt-postlink), the broken environment now works again.
I suspect something else I've done corrupted this folder. Possible a package I installed is the culprit? If I figure it out I'll post back.
I'm running Anaconda 3 with Python 3.6 on MacOS 10.12.
I created an environment using python 2.7 and opencv using conda create --name my-env python=2.7 anaconda(e.g. here). I activated the environment using source activate my-env and installed opencv. Both creation and activation seem to have worked: the anaconda3/envs/my-env folder exists and my terminal says (my-env) bob:~ alice'.
But how do I get Anaconda to use the new environment?
I tried:
starting the default Anaconda-Navigator.app, this just uses the default python path
changing the path in my ~/.bash-profile, but Anaconda still tries the default python path
starting the Anaconda-Navigator.app located in the my-env folder, fails with the OS error message "cannot be opened"
starting from terminal exits with the error message: FSPathMakeRef(~/anaconda3/envs/my-env/Anaconda-Navigator) failed with error -43.
Because I read somewhere that Anaconda 3 might not play nicely with Python 2.7, I tried the same workflow again, but creating an environment using python 3.5.4. Again to no avail.
What am I missing?
there a two possibilities to run a program in an env once the env has been created with conda:
Through default anaconda-navigator (the GUI of anaconda), there should be a menu on the left, with the following options: Home, Environments, Projects(beta),...
Simply klick on environments and choose/create a new one(see screenshot)
The "Applications on base(root)" can also be changed. If it does not display these options I recommend you to update conda.
Via terminal: activate the env as you mentioned earlier and then simply run the program you want, for example spyder, it should then use the right py version.
My default version of anaconda, the "root(base)" runs with python 2, therefore I added a few lines in my .bash_profile file to easily run python/spyder in either a default env or a specific one. maybe you can use something similar.
NONE='\033[00m'
YELLOW='\033[01;33m'
spy() {
if [[ "${1}" == "" ]]
then
ENV="py36env"
else
ENV=${1}
fi
source activate ${ENV} #enter env
echo -e "${YELLOW}CURRENT ENVIRONMENT: ${ENV} ${NONE}"
spyder #open spyder
echo -e "${YELLOW}EXIT ${ENV} ${NONE}"
source deactivate #exit env when spyder is quitted
}
You can now run spyder in the env "XXX" by entering "spy XXX" in the terminal. Or the default one by typing in "spy".
I am finding that when I create a Conda environment source activate environment and then deactivate that environment source deactivate environment, my native bash commands no longer work.
I've attached an image of ls working before enabling an environment and then not afterwards. I assume that this is something because of my PATH - but I am a novice and am not sure specifically what to fix. I am doing this in Git Bash on Windows. Anaconda 3 and Python 3.5.
Please work ls, come on
Most probably your conda environment is replacing the parameters and environment variables defined in your local bash profile. I am sure if you open a new environment(bash session), you will find everything is working fine.
I would suggest you add a command to reload bash profile after deactivating the environment, so that the profile variable values will again be applied. It should solve the problem.
There are a few bugs in the activate/deactivate scripts, which I've fixed in versions I call activate.cygwin and deactivate.cygwin.
activate.cygwin: fixes 2 places where the conda command returns a string with a return "\r" at the end that confounds processing.
deactivate.cygwin: fixes conversion of /cygdrive/... to avoid error
CondaValueError: Could not find environment: /cygdrivec:\Users\rjp\Anaconda2\envs...
I've posted the modified scripts to https://bitbucket.org/snippets/plevin/. See instructions at the top of activate.cygwin.
The correct way to deactivate a conda environment is to run conda deactivate, rather than source deactivate environment
Source: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html?highlight=activate#deactivating-an-environment
I use Git Bash in Windows 10 and encountered the same problem too. The previous answer does not work for me, probably because I don't have any path set in ~/.bash_profile and ~/.bashrc.
I checked my path variable in a fresh git bash, it looks like:
/c/Users/chen_x/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/mingw64/bin:/usr/bin:/c/Users/chen_x/bin....(more)
After . C:/Users/chen_x/Miniconda3/Scripts/activate C:/Users/chen_x/Miniconda3 and . activate snowflakes, the path becomes:
/C/Users/chen_x/Miniconda3/envs/snowflakes:(blalbla..):C:\Users\chen_x\Miniconda3\Library\bin;C:\Users\chen_x\Miniconda3;....(more)
I assume that git bash does not recognize pathes like C:\program files..., then added the following line to C:/Users/chen_x/Miniconda3/Scripts/activate right after the last EXPORT PATH= command (about line 78):
export PATH="$($_CONDA_PYTHON -c "import re; p=re.sub(r'\\\', r'/', r'$PATH'); p=re.sub('(;|:)([A-Z]):', lambda m: ':/'+m.group(2).lower(), p); print(p)")"
It works.
By the way, I create a ~/condaenv script to start conda environment:
#!/bin/bash
. C:/Users/chen_x/Miniconda3/Scripts/activate C:/Users/chen_x/Miniconda3
It would be convinent to :
. ~/condaenv
. activate snowflakes