I'm building a basic Twitter scraper with Python that I want to run off of my RaspPi 4b on an hourly basis. The script is written and works perfectly when called from the terminal using
python scraper.py
Now, I want to automate it to run without my own physical prompt. I did chmod with the script, then opened the crontab, and using the editor added this line (I understand that it's for every minute, I just want to see it work):
* * * * * /usr/bin/python home/pi/Desktop/twitter_scraper/scraper.py
However, nothing executes on its own. I'm not quite sure why this is because I specified the directories of both the Python program and the interpreter. Do I need to add anything else besides that line to the cron file? The Python script does access other files located in the same directory, but I did not think that would matter much. Do I need to restart my Pi for it to take effect?
When it come to python is better to run the code in to the directory where it is located. And in such case the cron will be something like:
* * * * * cd /home/pi/Desktop/twitter_scraper; /usr/bin/python scraper.py
you do not need anything more just do in this way
* * * * * python /home/pi/Desktop/twitter_scraper/scraper.py
if it doesn't work you can check your system log. and see the errors for debugging.
Related
I need to run a cron job on a python script to generate basemap plots.
The script by itself runs ok manually.
A simple print("Hello") at the start of the program with the rest commented out also runs ok on cron with
*/10 * * * * /usr/bin/python3 ~/PythonFiles/TestScript.py > /dev/null 2>&1 >>log.txt
I made the file an executable using chmod +x and added a shebang (#!/home/usr/anaconda3/bin/python) at the start of the program. I can monitor activity in the log file via a printed message at the start of the program too
When I come to run the "normal" program which includes modules (urllib.request, datetime, matplotlib, basemap, pygrib, numpy, ...), the script then stops outputting anything to log.txt
So I suspect it is to do with modules and possibly their locations. I checked and they seem to have been installed in various places (.../pkgs, .../conda-meta, .../site-packages, etc...)
First of all, is what I suspect correct?
Secondly, how do I fix it so that cron knows where to find all the libraries to run the job?
Many thanks!
I suspected it was to do with module location paths. After trawling through websites and tweaking inputs to cron, the following works!
SHELL=/bin/sh
HOME=/home/stephane
PYTHONPATH=/home/stephane/anaconda3/bin/python
PATH=/home/stephane/anaconda3/lib/python3.6/site-packages
*/2 * * * * /home/stephane/anaconda3/bin/python ~/PythonFiles/TestScript.py >/dev/null 2>&1 >> log.txt
Note: matplotlib seems to need "import matplotlib as mpl; mpl.use('Agg')" to run off cron.
Thanks to all!
I am attempting to run a python 3 script every 1 minute using cron on a raspberrypi 3, for testing, where eventually it will just be run once a day.
To start, I made a new cron job using: sudo crontab -e, and typed in the following code for a once a minute job:
*/1 * * * * /home/pi/folder/file.py
Then I saved and closed and waited. My python script emails me text when executed, so I should have seen an email come in. It runs fine (and emails me) when I execute it manually outside of cron.
So, what am I doing wrong with cron for it not to run? And do I need to make the python file executable or something with chmod?
Possible duplicate of Execute python Script on Crontab
EDIT:
Adding comment here since the comment box mangled my formatting.
In your example above it looks like you are just trying to "run" the file. You need to call the python executable, and pass it an argument that points to your file.
From the StackOverflow comment mentioned above look at this crontab entry:
*/2 * * * * /usr/bin/python /home/souza/Documets/Listener/listener.py
Take a look at the first part of the command /usr/bin/python this is pointing to the python executable not just to the .py file you want to run.
I'm rather green in python development but learning nevertheless.
I've written some python codes but they are mostly one-off use and to be run at command line. I still have no idea how to automate and schedule my codes. Lets say I have:
written a python script that pulls some CSV data from API and saves it in /tmp.
written another script to ingest the csv data and transform it into XML per line.
Each time I want to do this, I find myself doing:
$ python getdata.py
$ python converttoxml.py
In shell, I think one can write a wrapper script and cron it. Right? If so, how is this done in the world of python? Bear in mind we have to include all python libraries/ modules used, too.
P.S. developing in Linux environment using PyCharm.
Login to cron using crontab -e
Scroll to bottom at add a line following the following format:
m h dom mon dow command which is minutes, hours, day of month, day of week, command
So if you want to run your command on the hour every hour you would have
* /1 * * * * python path/to/file/getdata.py
whereas if you want it to run at 12 only then you would have
* 12 * * * * python path/to/file/getdata.py
Python files could be treated as executables. You just need to give them executing permissions with chmod +x my_file.py and tell the bash which interpreter should be used to parse the file by adding #!/usr/bin/python as the 1st line of the code file.
Once you've done the two things, you could just run your file using ./my_file.py and the python script should be executed.
From this point on, it's just like any ordinary program and could be used in cron/systemd/whatever other use you need. Once you have a working Python script, it should be treated no differently than any other executable on your system - in this regard, there's no "Python World".
As far as extra modules are concerned - this shouldn't be an issue as Python will still use the same way to resolve library location. It might be necessary to update $PYTHON_PATH to add a library path - but if the library is installed correctly it shouldn't be an issue while for libraries contained within your local directory (only in the case Python fails to find them) you could add:
sys.path.apped(os.path.dirname(__file__)) to the beginning of your script.
Recently I set up a twitter retweet bot, but after few minutes it's going down.
I used this python bot and I'm using "nohup python mrbot.py &" to keep it running in the background after closing the terminal. Even if I use it as "python mrbot.py", it stops working after around 30 minutes. So could someone please tell me how to automatically restart it every 30 minutes?
I'm a very beginner at this. *Remember I'm running it in background using "nohup". I don't know if this helps.
Crontab would be the ideal way to do it. Please search for crontab and how to enable and schedule a crontab job.
For your example a crontab entry would look like
30 * * * * /path/to/your/python/script.py
Since you are running ubuntu, you should have python in your path and have a shebang in the python script. A shebang is the first line in your python script with path to the python executable, it is in following format
#!/usr/bin/python
You are on Ubuntu, so you can use crontab command. Open terminal and type:
crontab -e
This will prompt you with (probably) an empty document. Then write the following line:
*/30 * * * * /usr/bin/python script.py
This will make sure your script runs every 30 minutes.
You can read more about crontab and what it does and how it works here
I know cron is capricious and I am trying to figure out how to deal with it on linux.
I have the following test_cron.py executable Python script for testing cron :
#!/usr/bin/env python
import os
os.makedirs('test_cron_dir')
f = open('test_cron_dir/test_file','w')
f.write('stuff')
f.close()
I added two lines to my crontab to run the script in two different folders :
* * * * * python /home/me/test_cron.py
* * * * * python /home/me/some_folder/test_cron.py
The problem is : cron runs the test_cron.py script located in /home/me/ but does not run the one located in /home/me/some_folder/.
I have changed the paths in the script to absolute paths but it does not change anything to the situation. Also, I have tried to use the root crontab and it does not change anything.
Can anyone please shine the light of knowledge and experience on me ? Thanks a lot.
cron is running crontab(5) entries from the home directory of the user.
You need to change appropriately the directory i.e. to call the chdir(2) syscall (either thru the cd shell builtin, or inside your python script using os.chdir).
You should query the current directory (using getcwd(3), or the pwd command, or the os.getcwd in Python) in your script.
Also check your PATH if running commands.