Develop Reference

AWS CodeDeploy error when trying to deploy python docker application in EC2

I am trying to deploy docker application, I am getting following error in Application Start phase:
[stderr]unable to prepare context: unable to evaluate symlinks in
Dockerfile path: lstat /opt/codedeploy-agent/Dockerfile: no such file
or directory
My appspec.yml file is as follows:
version: 0.0
os: linux
files:
- source: /
destination: /
file_exists_behavior: OVERWRITE
hooks:
ApplicationStop:
- location: scripts/kill_container.sh
timeout: 300
runas: ec2-user
BeforeInstall:
- location: scripts/install_dependencies.sh
timeout: 300
runas: ec2-user
ApplicationStart:
- location: scripts/start_container.sh
timeout: 300
runas: ec2-user
I have tried setting different destinations, as follows:
destination: /
destination: /home/ec2-user/Deployment
None of the above works for me.
My kill_container.sh code:
#!/usr/bin/env bash
set -e
echo "Stopping and removing the running container========"
docker rm -f python-api-docker || true
My install_dependencies.sh code:
#!/usr/bin/env bash
set -e
echo "====================================================="
echo $PWD
pwd
ls
My start_container.sh code:
#!/usr/bin/env bash
echo "Starting container==="
set -e
echo $PWD
pwd
ls
docker build -t python-api-docker .
docker run -p 5000:5000 python-api-docker
The docker file is as follows:
FROM python:3.9-alpine
EXPOSE 5000
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
WORKDIR /app
COPY . /app
# Creates a non-root user with an explicit UID and adds permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers
RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "api:app"]
I am attaching the error output:

certbot works in command line but fails when running through django call to script

So I have been dealing with this problem for days now and have exhausted every possible thing I can think of.
For clarification purposes I am running
Machine: Centos 7
WebServer: Django
Python Version: 3.6
I have this shell script at /usr/local/bin/activate_https_hostedvoice
#!/bin/bash
echo "<REMOVE_FOR_SECURITY>" | sudo -S curl -i -XPOST "http://<REMOVED_FOR_SECURITY>/records/""$1""/""$2"""
sleep 10
NGINX_CONFIG_FILE=/etc/nginx/guiconf.d/guiserver.conf
sed -i "s/]/,'""$1""']/g" /var/lib/guiserver/astgui2/db_pass.py
echo "<REMOVE_FOR_SECURITY>" | su -c "/sbin/service guiserver reload" root
echo "" >> ${NGINX_CONFIG_FILE}
echo "server{" >> ${NGINX_CONFIG_FILE}
echo "server_name $1;" >> ${NGINX_CONFIG_FILE}
echo "client_max_body_size 4G;" >> ${NGINX_CONFIG_FILE}
echo "include guiconf.d/server_params;" >> ${NGINX_CONFIG_FILE}
echo "include guiconf.d/static_files;" >> ${NGINX_CONFIG_FILE}
echo "include guiconf.d/web_path;" >> ${NGINX_CONFIG_FILE}
echo "include guiconf.d/local_path;" >> ${NGINX_CONFIG_FILE}
echo "include guiconf.d/ws_path;" >> ${NGINX_CONFIG_FILE}
echo "}" >> ${NGINX_CONFIG_FILE}
echo "<REMOVE_FOR_SECURITY>" | su -c "systemctl reload nginx" root
echo "<REMOVE_FOR_SECURITY>" | su -c "systemctl stop iptables" root
echo "<REMOVE_FOR_SECURITY>" | su -c "certbot --nginx -d ""$1"" --agree-tos --email <REMOVED_FOR_SECURITY> -n" root
echo "<REMOVE_FOR_SECURITY>" | su -c "systemctl reload nginx" root
echo "<REMOVE_FOR_SECURITY>" | su -c "systemctl start iptables" root
echo "<REMOVE_FOR_SECURITY>" | su -c "/sbin/service guiserver reload" root
echo "<REMOVE_FOR_SECURITY>" | su -c 'echo "0 */12 * * * certbot --nginx renew -n" >> /var/spool/cron/root' root
echo "<REMOVE_FOR_SECURITY>" | su -c "rm -f /var/lib/guiserver/bin/https_config" root
echo "<REMOVE_FOR_SECURITY>" | su -c "echo ""$1"" >> /var/lib/guiserver/bin/https_config" root
Whats happening here is first I am sending a curl request to add the A record to the nameserver.
Then I am adding the appropriate configurations for nginx
I then reload nginx and stop iptables so that certbot can reach the server
When I run this script in the command line bash /usr/local/bin/activate_https_hostedvoice <some_domain> <some_ip>
The script runs flawlessly and the A record is instantly pingable.
However when I run the script through django like so
os.system('echo "<REMOVE_FOR_SECURITY>" | su -c "bash /usr/local/bin/activate_https_hostedvoice ' + domain + ' ' + points_to + ' > /tmp/http_activate_log " root');
I can see that the A record is added to the nameserver and the nginx configuration values get added but when it comes to certbot this is the output that I receive
The following errors were reported by the server:
Domain: <REMOVED_FOR_SECURITY>
Type: dns
Detail: DNS problem: NXDOMAIN looking up A for <REMOVED_FOR_SECURITY>
- check that a DNS record exists for this domain
Upon the script exiting I then try to ping the A record and it is unpingable for several minutes on the server but if I try on my local computer or another computer and ping the domain its pingable
From the command line I have run over 30 test back to back and they have all gone through successful this issue only occurs when I call the script from django and it makes no sense to me
Update on the issue:
After further testing if I run curl -i -XPOST "http://<REMOVED_FOR_SECURITY>/records/<some_domain>/<some_ip>"
straight on the command line then it is not pingable instantly so I put it in a script and ran bash /tmp/test <some_domain> <some_ip>
Then the domain was pingable instantly
So naturally I changed the original script to be
bash /tmp/test ""$1"" ""$2""
instead of
echo "<REMOVE_FOR_SECURITY>" | sudo -S curl -i -XPOST "http://<REMOVED_FOR_SECURITY>/records/""$1""/""$2"""
But I am still having an issue with it not being pingable instantly
Thanks Joseph

i try this step and worked. just one line at a moment.
sudo pip uninstall requests
sudo pip uninstall urllib3
sudo yum remove python-urllib3 -y
sudo yum remove python-requests -y
sudo yum install python-urllib3 -y
sudo yum install python-requests -y
sudo yum install certbot python2-certbot-nginx
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Running poetry in jenkinsfile

Setup is Jenkins running in Kubernetes. I want to lint my code, run my tests, then build a container. Having trouble getting poetry to install/run in one of my build steps.
podTemplate(inheritFrom: 'k8s-slave', containers: [
containerTemplate(name: 'py38', image: 'python:3.8.4-slim-buster', ttyEnabled: true, command: 'cat')
])
{
node(POD_LABEL) {
stage('Checkout') {
checkout scm
sh 'ls -lah'
}
container('py38') {
stage('Poetry Configuration') {
sh 'apt-get update && apt-get install -y curl'
sh "curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python"
sh "$HOME/.poetry/bin/poetry install --no-root"
sh "$HOME/.poetry/bin/poetry shell --no-interaction"
}
stage('Lint') {
sh 'pre-commit install'
sh "pre-commit run --all"
}
}
}
}
Poetry install works fine, but when I go to activate the shell, it fails.
+ /root/.poetry/bin/poetry shell --no-interaction
Spawning shell within /root/.cache/pypoetry/virtualenvs/truveris-version-Zr2qBFRU-py3.8
[error]
(25, 'Inappropriate ioctl for device')

The issue here is that Jenkins runs a non-interactive shell and you are trying to start an interactive shell. The --no-interaction option doesn't mean a non-interactive shell but rather the shell not asking you questions:
-n (--no-interaction) Do not ask any interactive question
This answer explains it 🔑🔑.
I would just not call the shell and just use the poetry run 🏃🏃 command:
podTemplate(inheritFrom: 'k8s-slave', containers: [
containerTemplate(name: 'py38', image: 'python:3.8.4-slim-buster', ttyEnabled: true, command: 'cat')
])
{
node(POD_LABEL) {
stage('Checkout') {
checkout scm
sh 'ls -lah'
}
container('py38') {
stage('Poetry Configuration') {
sh 'apt-get update && apt-get install -y curl'
sh "curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python"
sh "$HOME/.poetry/bin/poetry install --no-root"
}
stage('Lint') {
sh "$HOME/.poetry/bin/poetry run 'pre-commit install'"
sh "$HOME/.poetry/bin/poetry run 'pre-commit run --all'"
}
}
}
}
✌️

Install poetry on container level and then parse the poetry.lock file with
poetry export --without-hashes --dev -f requirements.txt -o requirements.txt
Then install the dependencies with pip install -r requirements.txt INSTEAD of poetry install
Then you don't have to run your commands in the virtual env.

"mkvirtualenv command not found" within vagrantbox

I am trying to set up a django project using vagrant, but I am getting this error:
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-112-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
0 packages can be updated.
0 updates are security updates.
Last login: Sun Jan 28 08:21:28 2018 from 10.0.2.2
vagrant#ubuntu-xenial:~$ mkvirtualenv profiles_api --python=python3
mkvirtualenv: command not found
In my vagrant file I have the following :
sudo pip install virtualenvwrapper
if ! grep -q VIRTUALENV_ALREADY_ADDED /home/ubuntu/.bashrc; then
echo "# VIRTUALENV_ALREADY_ADDED" >> /home/ubuntu/.bashrc
echo "WORKON_HOME=~/.virtualenvs" >> /home/ubuntu/.bashrc
echo "PROJECT_HOME=/vagrant" >> /home/ubuntu/.bashrc
echo "source /usr/local/bin/virtualenvwrapper.sh" >> /home/ubuntu/.bashrc
fi
I have python 3.6,3.5 and anaconda installed, if that matters. Thank you for the help

Try this code in terminal when you entered into vagrant
export WORKON_HOME=~/Envs
mkdir -p $WORKON_HOME
source /usr/local/bin/virtualenvwrapper.sh
you have to change the path of file

First of all destroy the vagrant box by running following command:
vagrant destroy
Change your vagrant file to this:
sudo pip install virtualenvwrapper
if ! grep -q VIRTUALENV_ALREADY_ADDED /home/vagrant/.bashrc; then
echo "# VIRTUALENV_ALREADY_ADDED" >> /home/vagrant/.bashrc
echo "WORKON_HOME=~/.virtualenvs" >> /home/vagrant/.bashrc
echo "PROJECT_HOME=/vagrant" >> /home/vagrant/.bashrc
echo "source /usr/local/bin/virtualenvwrapper.sh" >> /home/vagrant/.bashrc
fi
Now initialize your vagrant box with vagrant init command and then run vagrant up command.

How to install lessc and nodejs in a Python virtualenv?

I would like to install a nodejs script (lessc) into a virtualenv.
How can I do that ?
Thanks
Natim

I like shorrty's answer, he recommended using nodeenv, see:
is there an virtual environment for node.js?
I followed this guide:
http://calvinx.com/2013/07/11/python-virtualenv-with-node-environment-via-nodeenv/
All I had to do myself was:
. ../bin/activate # switch to my Python virtualenv first
pip install nodeenv # then install nodeenv (nodeenv==0.7.1 was installed)
nodeenv --python-virtualenv # Use current python virtualenv
npm install -g less # install lessc in the virtualenv

Here is what I used so far, but it may be optimized I think.
Install nodejs
wget http://nodejs.org/dist/v0.6.8/node-v0.6.8.tar.gz
tar zxf node-v0.6.8.tar.gz
cd node-v0.6.8/
./configure --prefix=/absolute/path/to/the/virtualenv/
make
make install
Install npm (Node Package Manager)
/absolute/path/to/the/virtualenv/bin/activate
curl https://npmjs.org/install.sh | sh
Install lesscss
npm install less -g
When you activate your virtualenv you can use lessc

I created a bash script to automate Natim's solution.
Makes sure your Python virtualenv is active and just run the script. NodeJS, NPM and lessc will be downloaded and installed into your virtualenv.
http://pastebin.com/wKLWgatq
#!/bin/sh
#
# This script will download NodeJS, NPM and lessc, and install them into you Python
# virtualenv.
#
# Based on a post by Natim:
# http://stackoverflow.com/questions/8986709/how-to-install-lessc-and-nodejs-in-a-python-virtualenv
NODEJS="http://nodejs.org/dist/v0.8.3/node-v0.8.3.tar.gz"
# Check dependencies
for dep in gcc wget curl tar make; do
which $dep > /dev/null || (echo "ERROR: $dep not found"; exit 10)
done
# Must be run from virtual env
if [ "$VIRTUAL_ENV" = "" ]; then
echo "ERROR: you must activate the virtualenv first!"
exit 1
fi
echo "1) Installing nodejs in current virtual env"
echo
cd "$VIRTUAL_ENV"
# Create temp dir
if [ ! -d "tmp" ]; then
mkdir tmp
fi
cd tmp || (echo "ERROR: entering tmp directory failed"; exit 4)
echo -n "- Entered temp dir: "
pwd
# Download
fname=`basename "$NODEJS"`
if [ -f "$fname" ]; then
echo "- $fname already exists, not downloading"
else
echo "- Downloading $NODEJS"
wget "$NODEJS" || (echo "ERROR: download failed"; exit 2)
fi
echo "- Extracting"
tar -xvzf "$fname" || (echo "ERROR: tar failed"; exit 3)
cd `basename "$fname" .tar.gz` || (echo "ERROR: entering source directory failed"; exit 4)
echo "- Configure"
./configure --prefix="$VIRTUAL_ENV" || (echo "ERROR: configure failed"; exit 5)
echo "- Make"
make || (echo "ERROR: build failed"; exit 6)
echo "- Install "
make install || (echo "ERROR: install failed"; exit 7)
echo
echo "2) Installing npm"
echo
curl https://npmjs.org/install.sh | sh || (echo "ERROR: install failed"; exit 7)
echo
echo "3) Installing lessc with npm"
echo
npm install less -g || (echo "ERROR: lessc install failed"; exit 8)
echo "Congratulations! lessc is now installed in your virtualenv"

I will provide my generic solution to works with Gems and NPMs inside a virtualenv
Gems and Npm support to be customized via env settings : GEM_HOME and npm_config_prefix
You can stick the snippet below in your postactivate or activate script ( it s more matter if you use virtualenvwrapper or not)
export GEM_HOME="$VIRTUAL_ENV/lib/gems"
export GEM_PATH=""
PATH="$GEM_HOME/bin:$PATH"
export npm_config_prefix=$VIRTUAL_ENV
export PATH
Now , inside your virtualenv all libs installed via gem install or npm -g install will be installed in your virtualenv and binary added in your PATH
if you are using virtualenvwrapper you can make the change global to all your virtualenv if you modify the postactivate living inside your $VIRTUALENVWRAPPER_HOOK_DIR
This solution don't cover installing nodejs inside the virtualenv but I think that it s better to delegate this task to the packaging system (apt,yum,brew..) and install node and npm globally
Edit :
I recently created 2 plugin for virtualenvwrapper to do this automatically. There is one for gem and npm :
http://pypi.python.org/pypi/virtualenvwrapper.npm
http://pypi.python.org/pypi/virtualenvwrapper.gem