I understand that this question may have been repeated many times before - I am new to Nifi and unfortunately none of the previous answers worked for me.
I am trying to run a simple python script using the ExecuteStreamCommand processor (in Windows 10). However, I cannot call the python script directly from Nifi as it requires the conda environment to be activated (it gives me an import error for python packages- the same happens when the script is called from both nifi and command prompt).
I have since written a basic batch script to activate the conda environment and run the python script (see below).
activate.bat
CALL conda activate base
python sample.py
CALL conda deactivate
This works well and produces the output I need when run from the command line. However, calling the bat file from Nifi does not seem to do anything. It does not generate an error (simply pass the incoming flowfile to output with the working directory added to each line of the incoming csv file).
The config in Nifi:
Command Arguments: C:\Temp\activate.bat
Command Path: C:\Windows\System32\cmd.exe
Any help would be greatly appreciated.
This worked for me!
cmd "/c activate [my_env] && python my_script.py && deactivate"
Source: Conda environments and .BAT files
Still not sure what the underlying issue was ....
Related
I have a Python script that is designed to make some basic text edits and accepts command line parameters. The whole project is enclosed in a virtual environment too.
I am using python Click module to accept command line parameters, which does support auto completion in bash. I have a basic setup.py file to install the main script as a command or in a virtual environment by using pip install --editable ..
Now let's say main script is called edits. I should use _EDITS_COMPLETE=bash_source edits to generate a bash script to be run and enables auto completion. This does work fine in a virtualenv, where the script gets generated. However, outside of it, the normal script output is given.
In another words, in virtualenv the script returns the correct auto complete script, but outside of it, nothing happens.
So where may the issue with this be? I expect it to generate auto complete script outside virtualenv too.
Link to repository with the script I am trying to use: https://github.com/Astra3/DiscordText
Run pip install click (or pip install -r requirements.txt) in our host environment (outside the virtual env) and it will work.
So I have a python script that generates an animation - and it requires libraries that I have in a conda environment. I need to run this script as soon as my computer turns on, so I've written a short bash script that I added to "startup applications". This bash script runs on startup, and reads like this:
#!/bin/bash
conda activate myenv
cd ~/scripts
python generate.py
When I run this in terminal myself, it's fine, but whenever I turn on the computer, the python part of the script doesn't execute, and when I check errors i find:
conda: command not found
and then i also see the python script failed to run because it's missing libraries (from the conda environment not activating)
I have tried adding lines to the bash script replacing "conda activate" with "source activate", I have tried adding echo ". /home/<user>/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc to the bash script, replacing "conda" with /home/barrat/anaconda3/bin/conda, and even adding whoami to the bash script that runs at startup to make sure that i haven't magically become root by chance... none of this has worked. I would really appreciate any help. it's 3 AM and i'm a bit desperate.
You might have solved the issue yet, but for future viewers, this worked for me:
if [ -f "/path/to/anaconda3/etc/profile.d/conda.sh" ]; then
. "/path/to/anaconda3/etc/profile.d/conda.sh"
CONDA_CHANGEPS1=false conda activate myenv
fi
Add this instead of conda activate myenv.
Well as you are trying to activate an environment to start your scripts, it may also be possible for you to make a startup script itself to do the desired task by using subprocess module from python.
Try making a demo.py script like :
import os
import system
import subprocess
import x
subprocess.run(["command name", "value"]) #for all scripts you want to execute
and then you can put this python script to run at startup.
You can start quite a few amount of operations without noticable speed changes to your system and always can monitor it easily by starting the processes one after the other using time.sleep() in between two calls.
a colleague of mine has written a python script that I need to use, which is called within a shell script. It produces plots with matplotlib. However, when I try to run his script, it fails in matplotlib commands with "ImportError: No module named PyQt4". The python script is called within the shell script with a syntax like
./script.py
script.py begins with a line to specify the python exec to use from within his miniconda environment, like
#!/user/miniconda/envs/py27/bin/python
I think the problem is that the code uses the default PyQt on my system when I run this command. I tried running script.py with the python exec in his environment, but this gives the same error. This also occurs if I try to run the script on his computer when logged into my account. Is there a way that I can run this script as if I were my colleague within my account?
Have your colleague generate a yaml file with his environment dependencies, then create a copy of his environment on your computer to run the script.
# your coworker runs:
conda env export -n [name of his environment] > environ.yml
Once you get yaml file, you can run
conda env create -f environ.yml
to copy the environment. From there, activate it and run the script
# on Windows
activate [environment name]
python ./script.py
# on *nix
source activate [environment name]
python ./script.py
I have written my first python script using Spyder as the IDE within Anaconda. Yay! The script pulls data from google finance and emails it to me. When run inside Spyder it works fine.
Now I want to schedule that script to run at a specific time during the day. So, after researching, I have tried to set up the job to run in Cron with the following syntax:
15 12 * * * users/paul/desktop/pythonscript.py
I thought this would run the script at 12:15 but nothing happens.
I tried experimenting by opening the script in IDLE and running it or running it from a terminal but I can't get any of those to work because it tells me that none of the modules are imported.
So...can anybody tell me how to schedule a python script that was written in Spyder to run at a specific time?
Thanks!
You will need to specify the PATH variable within Cron and make sure python3 is in it. (It looks as from your comments you are using python3, make sure you know if you are using 2 or 3, just typing python will usually default to python 2) You can make normal edits like this with:
crontab -e
Then add the full path to python before your call to your job. EDIT: This path needs to be the path to your anaconda environment python (to avoid compatibility issues between other versions of python on your system).
PATH=path/to/anaconda/env/bin #you need to look this up
15 12 * * * python3 users/paul/desktop/pythonscript.py
See How to get CRON to call in the correct PATHs
If you don't include the folder that contains the anaconda environment python3 in your PATH, it will not run exactly like it does in spyder. If you would like to know where anaconda version of python is type this in bash:
conda info --envs
conda env list #or you could try this
If your command includes the call to your anaconda environment python, then you do not need the shebang in
pythonscript.py. If you don't want to include the call to python in your command in crontab, then include the shebang in your python script on the first line.
These may be useful:
run a crontab job using an annaconda env (see the second answer there)
https://conda.io/docs/user-guide/tasks/manage-environments.html (a guide for managing conda environments)
First check whether your script contains the python shebang or not.
#!/usr/local/bin/python
above will the the path for your python enviornment and this will run just fine or change your cron syntax, and tell it to call python to run your file and
use your complete path for your file such as /home/users/.....
15 12 * * * python users/paul/desktop/pythonscript.py
Goal - I created python scripts(Scipt X) using Spyder(conda base),I want to schedule it in cron.
Solution - I create one more script(script Y),written below code
prehand:-
switch to conda env by typing below command in terminal
conda activate //Switch to conda env
which python //check the python version in my case(~anaconda3/bin/python)
Code below in script Y:
import os,import subprocess
subprocess.call("~anaconda3/bin/python <FULL SCRIPT X PATH>")
I call this (script Y) in crontab.
Crontab entry
min hour date month week ~anaconda3/bin/python <full path to Script Y>
you are now ready to schedule spyder scripts in crontab.
I have a python file in which I'm using the subprocess module to execute some command line scripts.
I'm using Git bash to run this python file. In the file, I execute the script:
KG_URL=http://127.0.0.1:8900
This script sets the variable successfully when I run it manually on the git bash command line.
But when I execute this using the python file, it gives me the following error:
'KG_URL' is not recognized as an internal or external command,operable program or batch file.
I tried digging deeper into this and I found out that executing the python file on git bash is the equivalent of running those scripts on the Windows cmd. When I tried running the set command without parameters (to get the current environment variables) on the Windows cmd, I found out that the variable KG_URL does not exist. But when I ran the same command on git bash, I can see that KG_URL exists.
Any idea why this discrepancy exists? And how can I solve this issue?
The reason why I'm executing these scripts in a Python file is because I need to convert it into an exe later. Assuming that all environments where I run this exe will have git bash installed, is there any way of ensuring that these scripts run only through git bash, and not the Windows cmd?