I want to a up a cron job to run a python script each day within a virtual environment. So I've tried to set up a cron job but it does not seem to execute.
If I were to run the program from terminal normally, I would type:
source ig/venv/bin/activat enter to activate my virtual environment
cd ig/mybot/src/ navigate to my directory
python ultimate.py run my program
SO FAR this is my cron job. I've set it to 1 to run every minute just so I can see that it is working but nothing happens.
1 * * * * source ig/venv/bin/activate && cd ig/mybot/src/ && python ultimate.py
Edit: I have updated my program so no command line prompts are required. I am just needing to run these three simple commands.
You can wrap this up with another python script itself. Create a new python script and run it instead of cron.
Have a look into subprocess module.
Example:
Your command would become subprocess.call(['source','ig/venv/bin/activate'])
inside the wrapper python script.
Also, input("Enter the value") will prompt you for user input.
With the above two, your problem will be solved pythonically.
I'm not sure if it's a good idea, but you could do a script like this.
#!/usr/bin/env bash
PYTHON_PROJECT_DIR=/path/to/python/project/dir
pushd ${PYTHON_PROJECT_DIR}
VALUES="first line of stream\nsecondline of stream\n"
pipenv run /path/to/your/script.py < (echo -e $VALUES)
popd
pushd and popd are commands to move with the directory stack, so you'll be in the directory in the top of the stack, so by adding one directory, you move to the working directory, and by poping you'll get back to the initial position.
Using pipenv allows you to run the scripts in the virtual enviroment (It's not that hard to configure), that way you'll use the enviroment variables in the .env files for the project, and you'll only use the dependencies of this project. (python related).
If you pass the values like this, the python script when ever it requests a value from stdin it will use the values that you echoed (line by line, first line is first input and so on)
This could be a way.
Personally when ever I do cronjobs I like to run directly bash scripts, because, I could add extra logging, so having the wrapper script doesn't seem that unreasonable.
Another thing you could do, Is to get the python executable path (for the virtual enviroment), and use that as interpreter, by replacing the #!/usr/bin/env python to #!/path/to/pythons/virtual/env/interpreter
but you won't get the .env variables (may be there is a way to actually get them.
Related
Background:
I'm using Windows. I know some of programming with python. I don't know much about batch, which I think I might need to do what I want.
I will show a example so it becomes more clear what I'm trying to do.
Example:
When using git, after you install it, you can call the git command from anywhere of your computer, you can execute git commands, like git init and this will create a git file in your current folder.
I don't know exactly how git works or what language they use but I want to do the same thing, create my own command that I can execute from anywhere in my computer after I "install" my program.
What I'm trying to do:
I want to make my own command and when I call it, it executes a python script.
e.g.
I install my program and it creates a command called myprogram and when I type myprogram in the command line, it's like if I typed python myprogram.py. And myprogram -someargument would be the same as python myprogram.py -someargument.
What I tried until now:
I'm searched for How to make a environment variable that runs Python script? but I never get exactly what I want, is always something like How do I set environment variable using Python script?.
Maybe I'm making the wrong question and the result I want are not showing?
I'm looking for a explanation on how to do this or at least a tutorial/guide.
Edit 1:
As UnholySheep said in the comments, it's not environment variable, its commands, so I changed the question even does what I want to know is the same thing.
Files you need:
First you need a python script (obviously) so I created a file called myprogram.py that have this simple line:
print("This should be from a command")
After you need to make a batch file, in my case I used a .cmd file called myprogram.cmd that have:
#ECHO OFF
python_directory\python.exe python_script_directory\myprogram.py %*
Configurations to make:
You need to set in PATH environment variable the location of the batch file batch_file_directory\myprogram.cmd
And now if you execute in the command line myprogram it will print This should be from a command.
You can also use .exe or .bat files.
How can I create functions and software for my Linux server? Let me explain this in a bit more detail.
So for my Linux server which I access with my SSH client, I have made a few Python scripts that work fine, but what I really want to do is have these Python scripts active all the time, such that I can execute functions I've created in the script (such as "def time(): ...") just by typing "time" in to the command line rather than starting up a script with ./script-name.py and then type "time". Do I need to install my Python files in to the system in some way?
I struggled searching Google because I didn't fully understand what to search, and results that came up weren't really related to my request. I did find the cmd Python module and learned how to create cmd interpreters, however, in order for me to access the commands I defined in the cmd interpreter, I had to first start the script, which as I explained above, not what I am looking for.
How can I make script-level Python functions callable from the Linux command line?
If you're using Python, you'll still need to fire up the interpreter, but you can make that happen automatically.
Start by making your script executable. Run this command in the shell:
chmod +x script-name.py
ls -l script-name.py
The output of ls should look something like this (note the xs in the left-hand column):
-rwxr-xr-x 1 me me 4 Jan 14 11:02 script-name.py
Now add an interpreter directive line at the top of your script file - this tells the shell to run Python to interpret your script:
#!/usr/bin/python
Then add code at the end of the file that calls your function:
if __name__ == '__main__':
time()
The if statement checks to see if this is the file that is being executed. It means you can still import your module from another Python file without the time() function being automatically called, if you want.
Finally, you need to put your script in the executable path.
mkdir -p $HOME/bin
mv script-name.py $HOME/bin/
export PATH=$HOME/bin:$PATH
Now you should be able to run script-name.py, and you'll see the output of the time() function. You can rename your file to whatever you like; you can even remove the .py extension.
Additional things you can do:
Use the argparse module to add command line arguments, e.g. so you can run script-name.py time to execute the time() function
Put the script in a system-wide path, like /usr/local/bin, or
Add the export PATH=$HOME/bin:$PATH line to your .bashrc so that it happens by default when you log in
The answer above is by far more complete and more informative than mine. I just wanted to offer a quick and dirty alternative.
echo "alias time='<your script> time'" > ~/.bashrc
bash
Like I said, quick and dirty.
I am trying to run a Python script from cron. I am using crontab to run the command as a user instead of root. My Python script has the shebang at the top #! /usr/bin/env python and I did chmod +x it to make the script executable.
The script does work when running it from a shell but not when using crontab. I did check the /var/log/cron file and saw that the script runs but that absolutely nothing from stdout or stderr prints anywhere.
Finally, I made a small script that prints the date and a string and called that every minute that worked but this other script does not. I am not sure why I am getting these variable results...
Here is my entry in crontab
SHELL=/bin/bash
#min #hour day-of-month month day-of-week command
#-------------------------------------------------------------------------
*/5 * * * * /path/to/script/script.py
Here is the source code of my script that will not run from crontab but will run from the shell it is in when called like so ./script.py. The script is executable after I use the chmod +x command on it, by the way...:
#! /usr/bin/env python
#create a file and write to it
import time
def create_and_write():
with open("py-write-out.out", "a+") as w:
w.write("writing to file. hello from python. :D\n")
w.write("the time is: " + str(time.asctime()) + "\n")
w.write("--------------------------------\n")
def main():
create_and_write()
if __name__ == "__main__":
main()
EDIT
I figured out what was going wrong. I needed to put the absolute file paths in the script otherwise it would write to some directory I had not planned for.
Ok, I guess this thread is still going to help googlers.
I use a workaround to run python scripts with cron jobs. In fact, python scripts need to be handled with delicate care with cron job.
So I let a bash script take care of all of it.I create a bash script which has the command to run the python script. Then I schedule a cron job to run bash script. You can even use a redirector to log the output of bash command executing the python script.For example
#reboot /home/user/auto_delete.sh
auto_delete.sh may contain following lines:-
#!/bin/sh
echo $(date) >> bash_cron_log.txt
/usr/bin/python /home/user/auto_delete.py >> bash_cron_log.txt
So I don' need to worry about Cron Jobs crashing on python scripts.
You can resolve this yourself by following these steps:
Modify the cron as /path/to/script/script.py > /tmp/log 2> &1
Now, let the cron run
Now read the file /tmp/log
You will find out the reason of the issue you are facing, so that you can fix it.
In my experience, the issue is mostly with the environment.
In cron, the env variables are not set. So you may have to explicitly set the env for your script in cron.
I would like to emphasise one more thing. I was trying to run a python script in cron using the same trick (using shell script) and this time it didn't run. It was #reboot cron. So I used redirection in crontab as I mentioned in one of the above comments. And understood the problem.
I was using lot many file handlers in python script and cron runs the script from user's home directory. In that case python could not find the files used for file handlers and will crash.
What I used for troubleshooting was that I created a crontab as below. It would run the start.sh and throw all the stderror or stdoutput to file status.txt and I got error in status.txt saying that the file I used for file handler was not found. That was right because python script was executed by cron from user's home directory and then the script starts searching for files in home directory only.
#reboot /var/www/html/start.sh > /cronStatus/status.txt 2>&1
This will write everything happening during cron execution to status.txt file. You can see the error over there. I will again advice running python scripts using bash scripts for cronjobs. SOLUTION:- Either you use full path for all the files being used in script (which wasn't feasible for me, since I don't want script to be location dependent). Or you execute script from correct directory
So I created my cron as below-
#reboot cd /var/www/html/ && /var/www/html/start.sh
This cron will first change directory to correct location and then start the script. Now I don't have to worry about hardcoding full path for all files in script. Yeah, it may sound being lazy though ;)
And my start.sh looks like-
#!/bin/sh
/usr/bin/python /var/www/html/script.py
Hope it helps
Regards,
Kriss
I had a similar issue but with a little bit different scenario: I had a bash script (run.sh) and a Python script (hello.py). When I typed
sh run.sh
from the script directory, it worked; if I added the sh run.sh command in my crontab, it did not work.
To troubleshoot the issue I added the following line in my run.sh script:
printenv > env.txt
In this way you are able to see the environment variable used when you (or the crontab) run the script. By analyzing the differences between the env.txt generated from the manual run and the crontab run I noticed that the PWD variable was different. In my case I was able to resolve by adding the following line at the top of my .sh script:
cd /PYTHON_SCRIPT_ABSOLUTE_PATH/
Hope this could help you!
Another reason may be that the way we judged executed or not executed is wrong.
Right now I have a script which uses numpy that I want to run automatically on a server. When I ssh in and run it manually, it works fine. However, when I set it to run as a cron job, it can't find numpy. Apparently due to the shared server environment, the cron demon for whatever reason can't find numpy. I contacted the server host's tech support and they told me to set up a vps or get my own damn server. Is there any way to hack a workaround for this? Perhaps, by moving certain numpy files into the same directory as the script?
If you have numpy installed somewhere on the server, you can add it into the import path for python; at the beginning of your script, do something like this:
import sys
sys.path.append("/path/to/numpy")
import numpy
The cronjob runs with an empty environment. As such, it's either not using the same python binary as you are at the shell, or you have PYTHONPATH set, which it won't have under crontab.
You should run env -i HOME=$HOME sh to get a fascimile of the cronjob's environment. Set environment variables until your command works, and record them.
You can then set these in your crontab file, again using the env command, like:
* * * * * env PYTHONPATH=/my/pythonpath OTHERVAR=correct-value /path/to/mycommand
Processes invoked by the cron daemon have a minimal environment, generally consisting of $HOME, $LOGNAME and $SHELL.
It sounds like numpy is perhaps somewhere on your $PYTHONPATH? If so, you will need to specify that within the crontab line. Such as
/usr/bin/env PYTHONPATH=... <then the command to run>
If you are on a Linux system using vixie cron, then you can also specify global variables in your crontab by using lines such as
# my environment settings
PYTHONPATH = <path>
SOMETHING_ELSE = blah
<then my normal cron line>
See man -s 5 crontab
Your cron job is probably executing with a different python interpreter.
Log in as you (via ssh), and say which python. That will tell you where your python is. Then have your cron job execute that python interpreter to run your script, or chmod +x your script and put the path in a #! line at the top of the script.
I am trying to write what should be a super simple bash script. Basically activate a virtual env and than change to the working directory. A task i do a lot and condesing to one command just made sense.
Basically ...
#!/bin/bash
source /usr/local/turbogears/pps_beta/bin/activate
cd /usr/local/turbogears/pps_beta/src
However when it runs it just dumps back to the shell and i am still in the directory i ran the script from and the environment isn't activated.
All you need to do is to run your script with the source command. This is because the cd command is local to the shell that runs it. When you run a script directly, a new shell is executed which terminates when it reaches the script's end of file. By using the source command you tell the shell to directly execute the script's instructions.
The value of cd is local to the current script, which ends when you fall off the end of the file.
What you are trying to do is not "super simple" because you want to override this behavior.
Look at exec for replacing the current process with the process of your choice.
For feeding commands into an interactive Bash, look at the --rcfile option.
I imagine you wish your script to be dynamic, however, as a quick fix when working on a new system I create an alias.
begin i.e
the env is called 'py1' located at ~/envs/py1/ with a repository
location at ~/proj/py1/
alias py1='source ~/envs/py1/bin/activate; cd ~/proj/py1/;
end i.e
You can now access your project and virtualenv by typing py1 from anywhere in the CLI.
I know that this is no where near ideal, violates DRY, and many other programming concepts. It is just a quick and dirty way of getting your env and project accessible quickly without having to setup the variables.
I know that I'm late to the game here, but may I suggest using virtualenvwrapper? It provides a nice bash hook that appears to do exactly what you want.
Check out this tutorial: http://blog.fruiapps.com/2012/06/An-introductory-tutorial-to-python-virtualenv-and-virtualenvwrapper