I've been using GitHub actions to deploy a Python application to AWS Elastic beanstalk.
And I need to schedule a shell script from the source code to run it every minute.
I went through some AWS documents I found this doc about .ebextensions and that's my attempt:
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
* * * * * root /var/app/current/my-application/update_sha.sh
commands:
remove_old_cron:
command: "rm -f /etc/cron.d/mycron.bak"
But seems like it did not work. And I have no idea what I was wrong?
Related
Greetings fellow programmers,
I am currently using django-background-tasks(https://django-background-tasks.readthedocs.io/en/latest/)
to run some background tasks on AWS elasticbeanstalk. I initially use this command in the main .config container commands but I get timeout error in deployment because this management command will never finish(it continues to run on).
Now, I am trying using the approach suggested for running cron job on elasticbeanstalk(https://aws.amazon.com/premiumsupport/knowledge-center/cron-job-elastic-beanstalk/). Please take a look at my code, it is not running the command. what is wrong pls? I just need the command python manage.py process_tasks to keep running on. This is working properly on my local machine since i can easily open another terminal to fire up the python manage.py process_tasks command
files:
"/etc/cron.d/process_tasks_cron":
mode: "000644"
owner: root
group: root
content: |
* * * * * root /usr/local/bin/99_process_tasks.sh
"/usr/local/bin/99_process_tasks.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
date > /tmp/date
# Your actual script content
source /var/app/venv/*/bin/activate
python manage.py process_tasks
commands:
remove_old_cron:
command: "rm -f /etc/cron.d/*.bak"
this a working version of the code. thanks and hope it helps someone out there.
files:
"/etc/cron.d/cron_process":
mode: "000644"
owner: root
group: root
content: |
* * * * * root /usr/local/bin/task_process.sh
"/usr/local/bin/task_process.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
exec &>> /tmp/cron_capture_log.txt
date > /tmp/date
source /var/app/venv/staging-LQM1lest/bin/activate
cd /var/app/current
python manage.py process_tasks
commands:
remove_old_cron:
command: "rm -f /etc/cron.d/cron_process.bak"
I updated AWS platform from "Python 3.6 running on 64bit Amazon Linux" to "Python 3.7 running on 64bit Amazon Linux 2" and I'm not able to execute crons because I don't know how to add reference for Django environment properties there. Could you help me?
On the old platform everything works fine with this ".ebextensions/cron.config" setup:
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
3 * * * * root /usr/local/bin/script.sh > /home/ec2-user/script.log 2>&1
"/usr/local/bin/script.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
date > /tmp/date
# actual script content
source /opt/python/run/venv/bin/activate
source /opt/python/current/env
cd /opt/python/current/app/
python manage.py send_notification
exit 0
I know that I have to make changes on Linux 2 platform and I have to run activate -script from different location and this is what I have now:
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
3 * * * * root /usr/local/bin/script.sh > /home/ec2-user/script.log 2>&1
"/usr/local/bin/script.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
date > /tmp/date
# actual script content
source /var/app/venv/*/bin/activate
cd /var/app/current/
python manage.py send_notification
exit 0
I have also noticed that everything works fine when I try to do the same from .platform/hooks/predeploy and executing python from /var/app/staging, but that path is not available later on. The problem is that os.Environment - variables were not loaded from the EB configuration and I got KeyError from os.Environment['SOME_ENVIRONMENT_KEY'] reference.
If someone else is looking for resolution, I already resolved the problem with help of this document: https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-linux2/
Here is what worked for me:
files:
/usr/local/bin/my_cmd.sh:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
source /var/app/venv/*/bin/activate
cd /var/app/current
envvars="$(/opt/elasticbeanstalk/bin/get-config environment)"
# export env vars so they are available for the python management command
for s in $(echo $envvars | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" ); do
export $s
done
# check env var is there
env | grep RDS_HOSTNAME
python manage.py my_cmd
/etc/cron.d/my_cmd_cron:
mode: "000644"
owner: root
group: root
content: |
*/5 * * * * root /usr/local/bin/my_cmd.sh >> /home/ec2-user/logs/my_cmd.log 2>&1
commands:
rm_old_cron:
# remove old versions of the cron job on deloyment
command: "rm -fr /etc/cron.d/*.bak"
ignoreErrors: true
yum:
jq: []
Background:
Make env vars available to shell (didn't help for the cron job): https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-shell/
export env vars from json: https://stackoverflow.com/a/48513046
I'm working on implementing some cronjobs with my application running on elastic beanstalk but am unsure of how to proceed. My current cron-linux.config file in the .ebextension folder looks like:
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
* * * * * root /usr/bin/python opt/python/current/app/api/cron.py > /dev/null
commands:
remove_old_cron:
command: "rm -f /etc/cron.d/*.bak"
I've used eb ssh to make sure that the paths point to the correct location. The problem is that I'm not getting any error messages so it's quite hard to know where the problem is. Any help would be much appreciated!
You are supressing your outputs.
Try replacing the schedule line from:
* * * * * root /usr/bin/python opt/python/current/app/api/cron.py > /dev/null
To:
* * * * * root /usr/bin/python opt/python/current/app/api/cron.py > /home/<USER>/logs/backup.log 2>&1
You should be able to see the logs in /home/<USER>/logs/backup.log. Make sure that your script is outputing messages, success or error.
I am trying to schedule Python Script through Kubernetes CronJob but for some reason I am not able to understand how can I do it. I am able to run simple script like echo Hello World but that's not what I want
I tried using this specification:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: test
spec:
schedule: "*/1 * * * *"
concurrencyPolicy: "Forbid"
failedJobsHistoryLimit: 10
startingDeadlineSeconds: 600 # 10 min
jobTemplate:
spec:
backoffLimit: 0
activeDeadlineSeconds: 3300 # 55min
template:
spec:
containers:
- name: hello
image: python:3.6-slim
command: ["python"]
args: ["./main.py"]
restartPolicy: Never
But then I am not able to run it because main.py is not found, I understand that relative path is not supported so I hardcoded the path but then I am not able to find my home directory, I tried doing ls /home/ and over there my folder name is not visible so I am not able to access my project repository.
Initially I was planning to run bash script which can do:
Install requirements by pip install requirements.txt
Then run Python script
But I am not sure how can I do this with kubernetes, It is so confusing to me
In short I want to be able to run k8s CronJob which can run Python script by first installing requirements and then running it
where is the startup script ./main.py located? is it present in the image.
you need to build new image using python:3.6-slim as base image and add your python script to PATH. then you would be able to run it from k8s CronJob
I have a dockerfile which automates the building of an image.
I am using the docker cloud, connected to Digital Ocean as the server.
Within my dockerfile, I get the software I need, add the relevant GitHub repository containing the python scripts I wish to run. I then start the cron scheduler and add the script with appropriate times. For example:
The cron_files.txt file looks like this:
0 12 * * * /usr/bin/python /home/dir/run_check.py
0 15 * * * /usr/bin/python /home/dir/run_push.py
In my dockerfile, I do the following:
RUN service cron start
RUN service cron status
RUN crontab -u root cron_files.txt
In the log files, I can see that cron is succesfully started.
Edit, thanks to r0manarmy for this - How to run a cron job inside a docker container?
# Add crontab file in the cron directory
ADD crontab /etc/cron.d/hello-cron
# Give execution rights on the cron job
RUN chmod 0644 /etc/cron.d/hello-cron
# Create the log file to be able to run tail
RUN touch /var/log/cron.log
# Run the command on container startup
CMD cron && tail -f /var/log/cron.log
How do I edit the above to create the crontab file from the cron_files.txt rather than the above example?
I've tried ADD crontab cron_files.txt /etc/cron.d/feeds
But this returns:
ADD crontab cron_files.txt /etc/cron.d/feeds
lstat crontab: no such file or directory
Ps. I am using FROM debian:jessie
You probably want to set cron as the CMD:
In order to do this, just use the crond command on, say, alpine linux.
Take a look at this example for ideas:
https://gist.github.com/mhubig/a01276e17496e9fd6648cf426d9ceeec