I have done many research on the forum but I didn't find a clear answer to my question.
I have created a python code which extracts data from a website and put it in a SQL database (with SQLITE).
Now, I would like to make it run 24 hours a day on a server so that it collects data at each moment.
Although I know how to code, I don't have a broad knowledge of IT in general.
Would you know by chance how to solve my problem ?
Edit: I was more thinking about putting my scripts on a cloud server to make it run instead of running it directly from my pc.
Thank you very much.
Lcs
I can think of two options:
1) You use a simple "while true" and "wait" in python, so the script is running constantly.
import time
while True:
# Code
time.sleep(5) # goes to sleep for 5 seconds
You can also do this in shell script (infinite loop+sleep) and then run the program when you want.
2) You can use cronjobs to run the pytho script. You basically have the script the will scrap the website and it will be launched every X seconds/minutes/hours. You can read more about how to use cronjobs in Google
Depends on what system you are using (Unix or windows) you can use either:
1. Windows Task Scheduler for Windows
2. Crontab for Unix
Related
Background
I'm struggling to find a example of WDT in the way I want to use it. Wondering if I misunderstanding its use.
my python writing is pure hobby, honestly Classes intimidate me.
in short my program reads a number of sensors connected to a raspberry pi and writes the data to a cloud hosted object database.
i have an intermittent error that while I try to figure out I want to implement a based watchdog timer.
This is what I'd like to implement so in the very least I continue to collect and store data.
I've read about the builtin watchdog timer the raspberry pi has built in here: https://diode.io/raspberry%20pi/running-forever-with-the-raspberry-pi-hardware-watchdog-20202/
The problem I want the raspberry pi to reboot if my program hangs, but when that happens the OS is still fine, so the solution in the link above is not effective.
What I'd like to implement:
set the builtin watchdog timer to reboot the raspberry pi after 200 seconds without restarting (patting?) the timer. I think the instructions for this are in the link above.
Within my python script, after I iterate through each sensor, restart (or pat?) the watchdog timer and if 200 seconds elapse between pattings (meaning my program hangs) then RPi reboots.
is this possible?
can someone help me with some simple code? I was hoping to keep this simple and avoid classes and/or threads...
thank you in advance
The WDT is probably not the right solution for the problem you are describing. Based on your description, it sounds like what you have is a program that is intended to run periodically (either on a fixed schedule or in response to some event), and that program currently has a bug that is causing it to hang intermittently and never complete it's task or terminate.
The first and best way to solve that, I'm sure you can guess, is to fix the bug. But your thinking is not unreasonable either, and doing what you describe is very common. There are many valid approaches that will do what you want without the complexity of trying to deal with a hardware timer. One of the easiest is probably to just wrap the program in a shell script and use the timeout command to limit how long it is allowed to execute before it is terminated.
Assuming the script is located at /home/user/my_script.py, and you want to run it every 10 minutes, allowing it 2 minutes before it is killed, this would work:
create a wrapper shell script:
#!/usr/bin/env bash
if ! timeout 120s python /home/user/my_script.py; then
msg="$(date) - script exceeded timeout and was killed"
else
msg="$(date) - script ran successfully"
fi
echo "${msg}" >> /home/user/my_script.log
put the script in a file, say at /home/user/wrapper_script.sh and run chmod 755 /home/user/wrapper_script.sh to make it executable.
schedule the script to run every 10 minutes using cron. At a shell, use crontab -e to edit the users crontab file and add a new line like this:
*/10 * * * * /home/user/wrapper_script.sh
now, every 10 minutes the wrapper will start automatically, and it will kick off the python script. It will wait 2 minutes for the script to stop normally, after which it will reach the timeout and terminate it.
Note: depending on how your python program is written, you might have to use some other options to the timeout command to specify which signal it should use to stop the program. If it is a very basic python script, it should be fine using the default.
Edit: based on the comments, you might be able to just change the command you're using to this:
xterm -T "HMS" -geometry 100x70+10+35 -hold -e sudo timeout 120s /usr/bin/python3 /home/pi/h$
Doing that won't actually schedule the script to run at any fixed interval, so it assumes you already have something in place to handle that. All this will do is make srue that the script is restricted to 120 seconds of run time before it is killed.
I want to be able to run my python script which scans for something related to cryptocurrencies 1 minute after the same script has just been complete. Is this possible? Or could I maybe set a limit before the loop repeats itself or something?
The code I have isn't something I am willing to share due to its sensitive nature. It is a trading bot.
You have a few solutions:
Use a cron job if you are on a unix like platform (you can use this for the syntax, or man cron in the terminal to learn more about it)
Create a long running python script that sleeps for one minute before executing your logic again. Something like this:
import time
while True:
# execute code here
time.sleep(60)
If you are running it on Windows platform ,
You can create a batch file to run your script in cmd using the following command:
start "" "path_to_python.exe" "path_to_python_script"
then create a task in windows task scheduler .
You can refer
https://dev.to/tharindadilshan/running-a-python-script-every-x-minutes-in-windows-10-3nm9 . It might helps.
I wrote a Python script which I want to execute every 2 minutes. The script basically exports data and writes it into a .json file. Another script reads the data later and imports them into another system.
My question is, how can I execute the program every 2 minutes... I don't have any PHP skills, so I can't "translate" the program and create a simple cronjob on my webserver.
Is it possible with a Managed Server with Microsoft Server 2019?
Or is there any other possibility to do this?
Thank you in advance :-)
Use time module to sleep
import time
time.sleep(120)
I'm fairly competent with Python but I've never 'uploaded code' to a server before and have it run automatically.
I'm working on a project that would require some code to be running 24/7. At certain points of the day, if a criteria is met, a process is started. For example: a database may contain records of what time each user wants to receive a daily newsletter (for some subjective reason) - the code would at the right time of day send the newsletter to the correct person. But of course, all of this is running out on a Cloud server.
Any help would be appreciated - even correcting my entire formulation of the problem! If you know how to do this in any other language - please reply with your solutions!
Thanks!
Here are two approaches to this problem, both of which require shell access to the cloud server.
Write the program to handle the scheduling itself. For example, sleep and wake up every few miliseconds to perform the necessary checks. You would then transfer this file to the server using a tool like scp, login, and start it in the background using something like python myscript.py &.
Write the program to do a single run only, and use the scheduling tool cron to start it up every minute of the day.
Took a few days but I finally got a way to work this out. The most practical way to get this working is to use a VPS that runs the script. The confusing part of my code was that each user would activate the script at a different time for themselves. To do this, say at midnight, the VPS runs the python script (using scheduled tasking or something similar) and runs the script. the script would then pull times from a database and process the code at those times outlined.
Thanks for your time anyways!
I have written a function that crawls and clusters news articles. I want this function to restart every 10 minutes, aiming to get the newest articles. I wrote a Python script that this with the help of threading module:
import threading
def run():
do_it()
if not END:
threading.Timer(600.0, run).start()
END = False
threading.Timer(600.0, do_it).start()
It works fine with Python IDLE. Now I see in different forums that "crontab" is mentioned for this purpose. Since I plan to host this application on the web and since I have no prior experience with web development/hosting, what would you suggest to me: to continue with the previous code or to try to do it with the "crontab" function? I am working with the Django framework.
Using Django doesn't really work in this context. if you want to use a cron script then you will want it running independently of your webserver (you don't want your webserver ending after you finish the script). Alternatively you could start up a GET or POST request to an already running Django instance by this method.
First type crontab -e at a prompt in your unix based os or mac. (since you mentioned crontab I'll assume your not using windows)
This will allow you to edit a file for the user you are logged in as. This is the line you will want to type to get a do_it.py file to run every ten min.
*/10 * * * * do_it.py
For more information at your prompt type man 5 crontab. Other modifications you could make are prefixing the word sudo to the command which would allow you to change root's crontab.