So I have a python script that generates an animation - and it requires libraries that I have in a conda environment. I need to run this script as soon as my computer turns on, so I've written a short bash script that I added to "startup applications". This bash script runs on startup, and reads like this:
#!/bin/bash
conda activate myenv
cd ~/scripts
python generate.py
When I run this in terminal myself, it's fine, but whenever I turn on the computer, the python part of the script doesn't execute, and when I check errors i find:
conda: command not found
and then i also see the python script failed to run because it's missing libraries (from the conda environment not activating)
I have tried adding lines to the bash script replacing "conda activate" with "source activate", I have tried adding echo ". /home/<user>/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc to the bash script, replacing "conda" with /home/barrat/anaconda3/bin/conda, and even adding whoami to the bash script that runs at startup to make sure that i haven't magically become root by chance... none of this has worked. I would really appreciate any help. it's 3 AM and i'm a bit desperate.
You might have solved the issue yet, but for future viewers, this worked for me:
if [ -f "/path/to/anaconda3/etc/profile.d/conda.sh" ]; then
. "/path/to/anaconda3/etc/profile.d/conda.sh"
CONDA_CHANGEPS1=false conda activate myenv
fi
Add this instead of conda activate myenv.
Well as you are trying to activate an environment to start your scripts, it may also be possible for you to make a startup script itself to do the desired task by using subprocess module from python.
Try making a demo.py script like :
import os
import system
import subprocess
import x
subprocess.run(["command name", "value"]) #for all scripts you want to execute
and then you can put this python script to run at startup.
You can start quite a few amount of operations without noticable speed changes to your system and always can monitor it easily by starting the processes one after the other using time.sleep() in between two calls.
Related
I'm working with a Windows restricted by the company I work for using cdm.
I'm trying to activate a Python environment in a specific workspace, and for this I've created the following file named activatevenv2.bat:
cd venv2\Scripts\
echo "running activate"
activate.bat
cd ../..
The last line of this code means to come back two folders after executing the activate.bat file, but it's not working. Maybe because it's a Linux command? Using cd .. twice doesn't work either, in fact, I've tried several things without success.
Thanks a lot in advance
I understand that this question may have been repeated many times before - I am new to Nifi and unfortunately none of the previous answers worked for me.
I am trying to run a simple python script using the ExecuteStreamCommand processor (in Windows 10). However, I cannot call the python script directly from Nifi as it requires the conda environment to be activated (it gives me an import error for python packages- the same happens when the script is called from both nifi and command prompt).
I have since written a basic batch script to activate the conda environment and run the python script (see below).
activate.bat
CALL conda activate base
python sample.py
CALL conda deactivate
This works well and produces the output I need when run from the command line. However, calling the bat file from Nifi does not seem to do anything. It does not generate an error (simply pass the incoming flowfile to output with the working directory added to each line of the incoming csv file).
The config in Nifi:
Command Arguments: C:\Temp\activate.bat
Command Path: C:\Windows\System32\cmd.exe
Any help would be greatly appreciated.
This worked for me!
cmd "/c activate [my_env] && python my_script.py && deactivate"
Source: Conda environments and .BAT files
Still not sure what the underlying issue was ....
I'm trying to set up a job file to run on a linux cluster using
$ qsub network.job
I have loaded the module needed to execute the script, however a also need to activate a virtual environment that comes with the module.
#$ -S /bin/bash
#$ -cwd
#$ -j y
#$ -o output/
echo "Running job script"
module load python/python3
./network.py
echo "Finished job script"
Including
$source activate machinelearning
does not activate the virtualenv
What do I need to add to the file to activate the virtualenv machine learning that comes when loading the module python/python3?
source /PATH/WHERE/YOUR/VIRTUAL/ENVIRONMENT/EXISTS/bin/activate
for example in the subdirectory venv of your homedir
source ~/venv/machinelearning/bin/activate
Judging by your comment response, the source command may not be the problem. The typical approach in python virtual environments is to call source on the activate script inside the virtual environment. For more information on source, see this superuser post.
In this case, since the command source activate machinelearning works for you in an interactive session, it's likely that there's a script called activate in your home directory (or whichever directory you were working from) that takes machinelearning as an argument, and activates the appropriate virtual environment.
Here's my best stab at solving your immediate problem:
Since you are able to activate the virtual environment in an interactive session, log in to an interactive session and activate the virtual environment. Once you have done so, type which python3 in your terminal, and it should give you the full path to the virtual environment you're working in. Take that full path and replace the last python3 with activate, and that is the script you should source. For example:
$>which python3
/home/cmf05/environments/machinelearning/bin/python3
Then place this in your script between module load python/python3 and ./network.py:
source /home/cmf05/environments/machinelearning/bin/activate
As an aside: module load isn't something that's built in to any systems I've ever worked on, but it looks like you're submitting a job to a cluster that's managed by slurm or similar. It seems like this is loading all of the specific files you need for your compute job. I would recommend tagging the question with your specific cluster computing environment and putting that in the title for more focused help, in case this doesn't solve your issue.
When I'm trying to create a new Python 3 virtual environment by using mkvirtualenv (virtualenvwrapper command) and os.system like this
import os
os.system('mkvirtualenv foo')
nothing happens.
os.system("mate-terminal -e 'workon foo'")
doesn't work either.
The point is to quickly create a new virtual env and work on it later for each project (it's an automation script). virtualenvwrapper is the most convenient option.
The mkvirtualenv and workon commands are shell functions, not executables in your PATH[0].
To make them available in the shell you execute them in, you need to source the virtualenvwrapper.sh shell script defining them. You might be better off calling virtualenv /path/to/foo directly.
How to activate that virtualenv is another story, though, and will depend on the context you want to use it in. If you activate it in a subprocess, each process using it will have to be run in or under that child.
Hth,
dtk
PS In addition, you might look into the subprocess module (or even the third-party sh) for calling external programs. Happy coding :)
[0]: See $ which workon in a terminal vs $ which bash
The following codes in the bash shell script
env_name="<your env name>"
echo "Create virtual environment"
source `which virtualenvwrapper.sh`
mkvirtualenv $env_name -p python<$version>
source $HOME/.virtualenvs/$env_name/bin/activate
workon $env_name
then run bash script (for example: test.sh) from terminal source test.sh
a colleague of mine has written a python script that I need to use, which is called within a shell script. It produces plots with matplotlib. However, when I try to run his script, it fails in matplotlib commands with "ImportError: No module named PyQt4". The python script is called within the shell script with a syntax like
./script.py
script.py begins with a line to specify the python exec to use from within his miniconda environment, like
#!/user/miniconda/envs/py27/bin/python
I think the problem is that the code uses the default PyQt on my system when I run this command. I tried running script.py with the python exec in his environment, but this gives the same error. This also occurs if I try to run the script on his computer when logged into my account. Is there a way that I can run this script as if I were my colleague within my account?
Have your colleague generate a yaml file with his environment dependencies, then create a copy of his environment on your computer to run the script.
# your coworker runs:
conda env export -n [name of his environment] > environ.yml
Once you get yaml file, you can run
conda env create -f environ.yml
to copy the environment. From there, activate it and run the script
# on Windows
activate [environment name]
python ./script.py
# on *nix
source activate [environment name]
python ./script.py