Develop Reference

Scheduled for shell script on AWS Elastic beanstalk

I've been using GitHub actions to deploy a Python application to AWS Elastic beanstalk.
And I need to schedule a shell script from the source code to run it every minute.
I went through some AWS documents I found this doc about .ebextensions and that's my attempt:
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
* * * * * root /var/app/current/my-application/update_sha.sh
commands:
remove_old_cron:
command: "rm -f /etc/cron.d/mycron.bak"
But seems like it did not work. And I have no idea what I was wrong?

How to add environment prop references for cron.d on AWS EB Amazon Linux 2 platform

I updated AWS platform from "Python 3.6 running on 64bit Amazon Linux" to "Python 3.7 running on 64bit Amazon Linux 2" and I'm not able to execute crons because I don't know how to add reference for Django environment properties there. Could you help me?
On the old platform everything works fine with this ".ebextensions/cron.config" setup:
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
3 * * * * root /usr/local/bin/script.sh > /home/ec2-user/script.log 2>&1
"/usr/local/bin/script.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
date > /tmp/date
# actual script content
source /opt/python/run/venv/bin/activate
source /opt/python/current/env
cd /opt/python/current/app/
python manage.py send_notification
exit 0
I know that I have to make changes on Linux 2 platform and I have to run activate -script from different location and this is what I have now:
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
3 * * * * root /usr/local/bin/script.sh > /home/ec2-user/script.log 2>&1
"/usr/local/bin/script.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
date > /tmp/date
# actual script content
source /var/app/venv/*/bin/activate
cd /var/app/current/
python manage.py send_notification
exit 0
I have also noticed that everything works fine when I try to do the same from .platform/hooks/predeploy and executing python from /var/app/staging, but that path is not available later on. The problem is that os.Environment - variables were not loaded from the EB configuration and I got KeyError from os.Environment['SOME_ENVIRONMENT_KEY'] reference.

If someone else is looking for resolution, I already resolved the problem with help of this document: https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-linux2/

Here is what worked for me:
files:
/usr/local/bin/my_cmd.sh:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
source /var/app/venv/*/bin/activate
cd /var/app/current
envvars="$(/opt/elasticbeanstalk/bin/get-config environment)"
# export env vars so they are available for the python management command
for s in $(echo $envvars | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" ); do
export $s
done
# check env var is there
env | grep RDS_HOSTNAME
python manage.py my_cmd
/etc/cron.d/my_cmd_cron:
mode: "000644"
owner: root
group: root
content: |
*/5 * * * * root /usr/local/bin/my_cmd.sh >> /home/ec2-user/logs/my_cmd.log 2>&1
commands:
rm_old_cron:
# remove old versions of the cron job on deloyment
command: "rm -fr /etc/cron.d/*.bak"
ignoreErrors: true
yum:
jq: []
Background:
Make env vars available to shell (didn't help for the cron job): https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-shell/
export env vars from json: https://stackoverflow.com/a/48513046

Cronjob unable to recognize environment variables

I have a cronjob that runs every 5th minute which runs a bash script, which then runs a python file. All of this is running in a docker container:
crontab:
*/5 * * * * root ./rates/cmd.sh
cmd.sh:
#!/usr/bin/env bash
result=$(/usr/local/bin/python3 __main__.py --env $DB_ENVIRONMENT)
echo "$result" >> logs/"$(date +"%Y-%m-%d")".log;
main.py:
parser = argparse.ArgumentParser()
parser.add_argument("-e", "--env", dest="env", help="environments (choose 'dev' or 'live'); default is dev")
options = parser.parse_args()
print(options.env)
Docker Commands:
docker build -t rates .
docker run -d -e DB_ENVIRONMENT=dev rates
I am "sshing" into the docker container, and can verify that the DB_ENVIRONMENT variable has been set properly using "echo $DB_ENVIRONMENT".
The python script does not execute the print statement portion of the code, and I am assuming it fails on the parse_args() line. Nothing is outputted to the log file, when I am expecting the log to contain "dev".
When I run the bash script manually with ./cmd.sh or /usr/local/bin/python3 __main__.py --env $DB_ENVIRONMENT, the log file has "dev" in there / I can see output.
What is happening here?

Cron job with elastic beanstalk and django

I'm working on implementing some cronjobs with my application running on elastic beanstalk but am unsure of how to proceed. My current cron-linux.config file in the .ebextension folder looks like:
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
* * * * * root /usr/bin/python opt/python/current/app/api/cron.py > /dev/null
commands:
remove_old_cron:
command: "rm -f /etc/cron.d/*.bak"
I've used eb ssh to make sure that the paths point to the correct location. The problem is that I'm not getting any error messages so it's quite hard to know where the problem is. Any help would be much appreciated!

You are supressing your outputs.
Try replacing the schedule line from:
* * * * * root /usr/bin/python opt/python/current/app/api/cron.py > /dev/null
To:
* * * * * root /usr/bin/python opt/python/current/app/api/cron.py > /home/<USER>/logs/backup.log 2>&1
You should be able to see the logs in /home/<USER>/logs/backup.log. Make sure that your script is outputing messages, success or error.

Custom command on elastic beanstalk prompts invalid syntax in manage.py file

I've been trying to figure out how to best call a script with cronjobs and am unable to figure it out. Either I go with a custom command where I use the following in .ebextension/"some config file":
container_commands:
01_some_cron_job:
command: "cat .ebextensions/some_cron_job.txt > /etc/cron.d/mycron && chmod 644 /etc/cron.d/mycron"
leader_only: true
some_cron_job.txt:
* * * * * root source /opt/python/run/venv/bin/activate && source /opt/python/current/env && /usr/bin/python /opt/python/current/app/manage.py cron_command >> /var/log/myjob.log 2>&1
This works when i run the command locally but after having uploaded it to eb I get the following error:
File "/opt/python/current/app/manage.py", line 18
) from exc
^ SyntaxError: invalid syntax
Or I could call the script directly:
* * * * * root source /opt/python/run/venv/bin/activate && source /opt/python/current/env && /usr/bin/python /opt/python/current/app/api/cron.py >> /var/log/myjob.log 2>&1
But am then getting import errors when trying to import a function from a another file in the same directory:
ImportError: attempted relative import with no known parent package
I'm quite lost and would appreciate any help.

I managed to find a working solution where I instead used:
files:
"/etc/cron.d/mycron":
mode: "000644"
owner: root
group: root
content: |
0/10 * * * * root source /opt/python/current/env && /opt/python/run/venv/bin/python3 /opt/python/current/app/manage.py cron_command >> /var/log/newjob.log 2>&1
commands:
remove_old_cron:
command: "rm -f /etc/cron.d/mycron.bak"
I think the problem came about because of some python version problems in the virtual environment.