Develop Reference

Running a python script at a fixed time everyday with crontab

First of all, I have tried every method suggested on this site and others but I still can't get it to work.
My python script is located in my home folder. It imports modules like requests, time, and other third party modules. It works fine with command line.
But it doesn't work with crontab. I think the problem is that when it's run from cron, import doesn't work and the script fails.
crontab:
PATH=/sbin:/bin:/usr/sbin:/usr/bin
SHELL=/bin/bash
PYTHONPATH=/home/chanzerre:/usr/lib64/python35.zip:/usr/lib64/python3.5:/usr/lib64/python3.5/plat-linux:/usr/lib64/python3.5/lib-dynload:/usr/lib64/python3.5/site-packages:/usr/lib/python3.5/site-packages
* * * * * /home/chanzerre/script.py
My Python script's structure:
#! /usr/bin/python3.5
import requests as req
import time
from pprint import pprint
# third party imports here
#code here
Can anyone help?
Would give more details if needed.
P.S.
Please, don't mark it as dupe, because I have tried all methods suggested in the similar questions asked by others and for the love of my life, it still doesn't work.

Have you tried making your script.py an executable file? Just typing the path to a Python file won't run the script. You need to add a shebang to your file:
#!/usr/bin/python
and then make it executable:
$ mv /home/chanzerre/script.py /home/chanzerre/script
$ chmod +x /home/chanzerre/script
then your crontab becomes
* * * * * /home/chanzerre/script
Try that and see if it runs your file.

Have you set all the environment variables that your script needs in order to run? If not, it won't work.For example,if you are running that script behind a proxy server, and the proxy variable is not set, it won't work.
Set your proxy variable in crontab or in the python script itself and it will work.

In cron jobs (Python), what does the -m flag stand for and how is it used?

I am trying to set up a cron job which is executing a python file of mine. The python file is using some manually installed modules. The cron job now throws an error, as it 'cannot find' the specified module (yes, I tested it: if executed manually the script does work & have access to the module).
I did now recieve the cryptic info (from the hoster’s support) to 'try adding the -m flag to the command, followed by the path to the module that it cannot find.' Unfortunatelly I do not quite understand this advice.
Assuming that my cron job command (via Cpanel) would out of the box be:
0 * * * * python /home/public_html/cgi-bin/cronrun.py
which works if the python script does not rely on external modules.
So my questions are:
Is the -m flag appropriate?
If so, how do I use it?
And what do I do, if there is more than just one additional module that the script needs?
Thank you very much in advance!

Your cron job won't likely be running with the same environment that you have. To see this, first run env > [somepath_that_you_can_reach]. Then set up a cron to do the same thing in a shell script using a different path. Compare the two. You will need your PYTHONPATH to be the same for the cron job to work. If that is the problem, then in your python script:
import sys
sys.path.append('[the path part that you need for it to work]')
before your import statements.

Python Script not running in cron

I am trying to run a Python script from cron. I am using crontab to run the command as a user instead of root. My Python script has the shebang at the top #! /usr/bin/env python and I did chmod +x it to make the script executable.
The script does work when running it from a shell but not when using crontab. I did check the /var/log/cron file and saw that the script runs but that absolutely nothing from stdout or stderr prints anywhere.
Finally, I made a small script that prints the date and a string and called that every minute that worked but this other script does not. I am not sure why I am getting these variable results...
Here is my entry in crontab
SHELL=/bin/bash
#min #hour day-of-month month day-of-week command
#-------------------------------------------------------------------------
*/5 * * * * /path/to/script/script.py
Here is the source code of my script that will not run from crontab but will run from the shell it is in when called like so ./script.py. The script is executable after I use the chmod +x command on it, by the way...:
#! /usr/bin/env python
#create a file and write to it
import time
def create_and_write():
with open("py-write-out.out", "a+") as w:
w.write("writing to file. hello from python. :D\n")
w.write("the time is: " + str(time.asctime()) + "\n")
w.write("--------------------------------\n")
def main():
create_and_write()
if __name__ == "__main__":
main()
EDIT
I figured out what was going wrong. I needed to put the absolute file paths in the script otherwise it would write to some directory I had not planned for.

Ok, I guess this thread is still going to help googlers.
I use a workaround to run python scripts with cron jobs. In fact, python scripts need to be handled with delicate care with cron job.
So I let a bash script take care of all of it.I create a bash script which has the command to run the python script. Then I schedule a cron job to run bash script. You can even use a redirector to log the output of bash command executing the python script.For example
#reboot /home/user/auto_delete.sh
auto_delete.sh may contain following lines:-
#!/bin/sh
echo $(date) >> bash_cron_log.txt
/usr/bin/python /home/user/auto_delete.py >> bash_cron_log.txt
So I don' need to worry about Cron Jobs crashing on python scripts.

You can resolve this yourself by following these steps:
Modify the cron as /path/to/script/script.py > /tmp/log 2> &1
Now, let the cron run
Now read the file /tmp/log
You will find out the reason of the issue you are facing, so that you can fix it.
In my experience, the issue is mostly with the environment.
In cron, the env variables are not set. So you may have to explicitly set the env for your script in cron.

I would like to emphasise one more thing. I was trying to run a python script in cron using the same trick (using shell script) and this time it didn't run. It was #reboot cron. So I used redirection in crontab as I mentioned in one of the above comments. And understood the problem.
I was using lot many file handlers in python script and cron runs the script from user's home directory. In that case python could not find the files used for file handlers and will crash.
What I used for troubleshooting was that I created a crontab as below. It would run the start.sh and throw all the stderror or stdoutput to file status.txt and I got error in status.txt saying that the file I used for file handler was not found. That was right because python script was executed by cron from user's home directory and then the script starts searching for files in home directory only.
#reboot /var/www/html/start.sh > /cronStatus/status.txt 2>&1
This will write everything happening during cron execution to status.txt file. You can see the error over there. I will again advice running python scripts using bash scripts for cronjobs. SOLUTION:- Either you use full path for all the files being used in script (which wasn't feasible for me, since I don't want script to be location dependent). Or you execute script from correct directory
So I created my cron as below-
#reboot cd /var/www/html/ && /var/www/html/start.sh
This cron will first change directory to correct location and then start the script. Now I don't have to worry about hardcoding full path for all files in script. Yeah, it may sound being lazy though ;)
And my start.sh looks like-
#!/bin/sh
/usr/bin/python /var/www/html/script.py
Hope it helps
Regards,
Kriss

I had a similar issue but with a little bit different scenario: I had a bash script (run.sh) and a Python script (hello.py). When I typed
sh run.sh
from the script directory, it worked; if I added the sh run.sh command in my crontab, it did not work.
To troubleshoot the issue I added the following line in my run.sh script:
printenv > env.txt
In this way you are able to see the environment variable used when you (or the crontab) run the script. By analyzing the differences between the env.txt generated from the manual run and the crontab run I noticed that the PWD variable was different. In my case I was able to resolve by adding the following line at the top of my .sh script:
cd /PYTHON_SCRIPT_ABSOLUTE_PATH/
Hope this could help you!

Another reason may be that the way we judged executed or not executed is wrong.

Cron not running Python script if script not in home folder

I know cron is capricious and I am trying to figure out how to deal with it on linux.
I have the following test_cron.py executable Python script for testing cron :
#!/usr/bin/env python
import os
os.makedirs('test_cron_dir')
f = open('test_cron_dir/test_file','w')
f.write('stuff')
f.close()
I added two lines to my crontab to run the script in two different folders :
* * * * * python /home/me/test_cron.py
* * * * * python /home/me/some_folder/test_cron.py
The problem is : cron runs the test_cron.py script located in /home/me/ but does not run the one located in /home/me/some_folder/.
I have changed the paths in the script to absolute paths but it does not change anything to the situation. Also, I have tried to use the root crontab and it does not change anything.
Can anyone please shine the light of knowledge and experience on me ? Thanks a lot.

cron is running crontab(5) entries from the home directory of the user.
You need to change appropriately the directory i.e. to call the chdir(2) syscall (either thru the cd shell builtin, or inside your python script using os.chdir).
You should query the current directory (using getcwd(3), or the pwd command, or the os.getcwd in Python) in your script.
Also check your PATH if running commands.

Workaround Way To Install Numpy?

Right now I have a script which uses numpy that I want to run automatically on a server. When I ssh in and run it manually, it works fine. However, when I set it to run as a cron job, it can't find numpy. Apparently due to the shared server environment, the cron demon for whatever reason can't find numpy. I contacted the server host's tech support and they told me to set up a vps or get my own damn server. Is there any way to hack a workaround for this? Perhaps, by moving certain numpy files into the same directory as the script?

If you have numpy installed somewhere on the server, you can add it into the import path for python; at the beginning of your script, do something like this:
import sys
sys.path.append("/path/to/numpy")
import numpy

The cronjob runs with an empty environment. As such, it's either not using the same python binary as you are at the shell, or you have PYTHONPATH set, which it won't have under crontab.
You should run env -i HOME=$HOME sh to get a fascimile of the cronjob's environment. Set environment variables until your command works, and record them.
You can then set these in your crontab file, again using the env command, like:
* * * * * env PYTHONPATH=/my/pythonpath OTHERVAR=correct-value /path/to/mycommand

Processes invoked by the cron daemon have a minimal environment, generally consisting of $HOME, $LOGNAME and $SHELL.
It sounds like numpy is perhaps somewhere on your $PYTHONPATH? If so, you will need to specify that within the crontab line. Such as
/usr/bin/env PYTHONPATH=... <then the command to run>
If you are on a Linux system using vixie cron, then you can also specify global variables in your crontab by using lines such as
# my environment settings
PYTHONPATH = <path>
SOMETHING_ELSE = blah
<then my normal cron line>
See man -s 5 crontab

Your cron job is probably executing with a different python interpreter.
Log in as you (via ssh), and say which python. That will tell you where your python is. Then have your cron job execute that python interpreter to run your script, or chmod +x your script and put the path in a #! line at the top of the script.