This question already has answers here:
How to execute script on schedule?
(3 answers)
Closed 1 year ago.
I have my python scripts which execute every weekday and refreshes tableau datasets. This is all being done incremental refresh on a daily basis.
Now, during weekends instead of manual refresh, I want these scripts to run with a full refresh. Any ideas on how we can achieve this?
Example:
I have tasks in my script like
#Run Tasks
refresh_tableA()
refresh_tableB()
refresh_tableC()
refresh_tablemain(i)
refresh_tableD()
Where the refresh_maintable() is an incremental refresh. Want that to trigger to a full refresh on weekends.
You need to have two cron entries.
week days only - call the python code and order it to create partial refresh
wekends - call the python code and order it to create full refresh
I think what you are looking for is a scheduler.
You have 2 options here either use Task Scheduler/Cronjob (Based on your OS) or using python library for this.
I think there is library for cronjob in python pycron.
That way you can schedule you scripts easily. let me know if you still dont understand the flow or want code. Most probably you'll find something on github.
Related
I was able to write a script to scrape using selenium, right now I'm trying to automate it so it can work periodically on a server so I don't bother myself by running it from my local, I did a lot of googling but I got no clue of how I can do that, can anyone simplify things for me ..
In order to run a python script on a linux server periodically you can make a cronjob, since you already have a python script which most probably fetches or scrapes data and saves it in a file. You can make a cronjob and set the exact time it has to run, say for instance after every 2 hours you can do it using something like this,
crontab -e
this will open a editor in your terminal, at the bottom of the text just write timing and the command to be executed.
* * * * */path/to/your/code.py
from this link you can find out how to fill out the stars https://crontab.guru/#*_*_*_*_1
if you need anymore help with using cronjobs take a look at this https://www.geeksforgeeks.org/scheduling-python-scripts-on-linux/
You can simply redo your script on pythonanywhere and schedule it as a task and choose the frequency that you want the script to be executed. The current frequency options available include; script runs always, hourly or daily.
i dont know if it'll really work but
while True:
whole
code
here
time.sleep(period required)
This question already has answers here:
How to use python to schedule tasks in a Django application
(2 answers)
Closed 9 months ago.
I want to make a website where every month a fixed amount will increase in student's database. I want that i'll write a function and the function will run every first day of a month
I Want to Add a certain amount of fees to be added in every individual student account every month.
To do that, I need to run a function in the 1st day of month. How do I run a function continuously after every month?
Creating a function that's triggered on an hourly basis would be a good idea. Scheduled jobs/tasks, like adding monthly fees, are good candidates for this.
You could also look into cron jobs, which would be useful if you're on a Linux server, or task scheduler for Windows.
Here's a decent tutorial on how to make timer triggered cloud functions with AWS:
https://docs.aws.amazon.com/lambda/latest/dg/services-cloudwatchevents-tutorial.html
I Found a solution by searching here and there...
I need a package to be installed and then that'll do the job!!
Here's the link for that answer.
I am new to python.
I created a simple Selenium bot that basically runs on a while loop.
However, it runs on an external machine that I do not have access at all times.
It runs into unexpected crashes sometimes, and it takes hours and sometimes days until I check it again only to find a crashed bot. With that said, I'd like to know if a monitoring module or program I can implement into my bot, so I can easily check if the program is running or not.
Some have suggested a WebSocket heartbeat, however, I do not know, after extensive research, how I would go about implementing such an intricate system. Thank you for taking the time to read my question.
Any answers appreciated!
You can just put a few lines of code at the appropriate point in your script (after the Selenium bot completes its work successfully) to write the current date and time into a file.
A second script can be scheduled just to read that file and ring an alarm bell (e.g. send you an email) if it is X hours since the timestamp was last updated.
Alternatively you could write a timestamp to a database table or use whatever other shared systems you have.
I have a Python bot running PRAW for Reddit. It is open source and thus users could schedule this bot to run at any frequency (e.g. using cron). It could run every 10 minutes, or every 6 hours.
I have a specific function (let's call it check_logs) in this bot that should not run every execution of this bot, but rather only once a day. The bot does not have a database.
Is there a way to accomplish this in Python without external databases/files?
Generally speaking, it's better (and easier) to use the external database or file. But, if you absolutely need it you could also:
Modify the script itself, e.g. store the date of the last run in commented out last line of the script.
Store the date of the last update on the web, for example, in your case it could be a Reddit post or google doc or draft email or a site like Pastebin, etc.
Change the "modified date" of the script itself and use it as a reference.
If you're using cron you can run it with command line arguments.
And define in cron eg. python3 main.py daily for the daily run that you need and python3 main.py frequent for the other version.
I'm doing it that way and it worked optimally by now.
I would like to recreate some data in my project every 30 minutes (prices that change). also I got another job that needs to refresh every minute.
Now I heard I should use a daemon. but I'm not sure how that works.
Can someone put me into the right direction.
Also should i make an extra model to save that temporary data or is that part of the daemon?
PS: not sure if stack overflow can be used for this sort of questions, but i don't know where to search for this sort of information
You don't want a daemon. You just want cron jobs.
The best thing to do is to write your scripts as custom Django management commands and use cron to trigger them to run at the specified intervals.