Develop Reference

Run Python script from browser using virtual environment on EC2 instance

TLDR;
I created a virtual environment on my EC2 instance. How can I access this from the browser?
Hey everyone,
I created a virtual environment, following this tutorial, on my EC2 instance to run a simple Python script. Within the terminal, it works without errors. However, I have made a web application and I would like to activate this script from the browser using the virtual environment. When I try this, I get a "Permission denied" error.
PHP
$output=shell_exec('bash /var/app/current/scripts/script.sh');
echo "<pre>$output</pre>";
script.sh
#!/bin/bash
source /home/ec2-user/venv/python3/bin/activate
python3 /var/app/current/scripts/test.py
test.py
from datetime import datetime
from bs4 import BeautifulSoup
import requests
print('hello')
print(datetime.now())
url = "https://www.stackoverflow.com/"
website = requests.get(url).text
soup = BeautifulSoup(website, "html.parser")
print(soup.title)
error
/var/app/current/scripts/script.sh: line 2: /home/ec2-user/venv/python3/bin/activate: Permission denied
Traceback (most recent call last):
File "/var/app/current/scripts/test.py", line 2, in <module>
from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'
What I have tried:
I tried to change the permissions on the virtual environment using the following:
chmod a+x /home/ec2-user/venv
This should give all users access to the virtual environment folder: /home/ec2-user/venv
However, I am still getting the error:
/home/ec2-user/venv/python3/bin/activate: Permission denied
I have also tried to give all users the possibility of executing the activation script (/home/ec2-user/venv/python3/bin/activate):
chmod 665 /home/ec2-user/venv/python3/bin/activate
Which results in:
-rw-rw-r-x 1 ec2-user ec2-user /home/ec2-user/venv/python3/bin/activate
However, I still get the same error:
/home/ec2-user/venv/python3/bin/activate: Permission denied
Note:
Note that if I only import datetime and I comment out bs4 and requests (along with everything else regarding BeautifulSoup), then the script works great as it does not have to access the virtual environment to pull in the packages.
*Virtual environment tutorial

You get this error because you have not added libraries that are used in the python script to the virtual env.
In the tutorial you mentioned only boto library is installed.
You need to install libraries you use.
Run this from the command line:
source /home/ec2-user/venv/python3/bin/activate
pip install beautifulsoup4
pip install requests
Alternatively you can create a file and name it for example /home/ec2-user/requirements.txt for example and list all requirements your script use:
beautifulsoup4
requests
Then you can use this file to install all requirements into virtual env:
source /home/ec2-user/venv/python3/bin/activate
pip install -r /home/ec2-user/requirements.txt

Solved!
I got some help from this post, however, needed to modify a few things.
Let's dive into what his answer was:
sudo chown -R your_username:your_username path/to/virtuaelenv/
Okay, this is great, but I needed a bit of information.
For me, the web application's username is webapp.
Then, one thing that isn't very clear above is the path. So, my path is:
/home/ec2-user/venv/python3/bin/activate
as mentioned above. Here, you need to change permissions to the /home/ec2-user and NOT to /home/ec2-user/venv
So, to give my application permission to my virtual environment, I needed ran:
sudo chown -R webapp:webapp /home/ec2-user
That worked in the browser! However, this took away my ability to work with it on the server. To do so, I would have to switch it back to:
sudo chown -R ec2-user:ec2-user /home/ec2-user
Being far from ideal to switch back and forth, I tried to change the permissions with chmod instead.
sudo chmod 711 /home/ec2-user
Now I have read, write, and execution permissions whereas everyone else, including the web app, can only execute.
Now it all works 🤓

ElasticBeanstalk suddenly starts failing to deploy Django App with "Cannot use ImageField because Pillow is not installed" exception

I am using elastic beanstalk to deploy my Django application. Today it suddenly stopped working without any breaking changes from the application side (I've changed some templates, nothing more).
The deployment time outs after 10 minutes of trying to deploy the app and nothing happens.
The only more or less useful hints I can see in the log is this:
[2020-02-20T15:00:20.437Z] INFO [19057] - [Application update .../postbuild_0_myproject/Command 01_migrate] : Activity execution failed, because: SystemCheckError: System check identified some issues:
ERRORS:
education.Author.photo: (fields.E210) Cannot use ImageField because Pillow is not installed.
HINT: Get Pillow at https://pypi.org/project/Pillow/ or run command "pip install Pillow".
education.Course.cover_image: (fields.E210) Cannot use ImageField because Pillow is not installed.
HINT: Get Pillow at https://pypi.org/project/Pillow/ or run command "pip install Pillow".
education.CourseCategory.icon_image: (fields.E210) Cannot use ImageField because Pillow is not installed.
HINT: Get Pillow at https://pypi.org/project/Pillow/ or run command "pip install Pillow".
Using staging settings
App receivers connected
(ElasticBeanstalk::ExternalInvocationError)
[2020-02-20T15:00:20.437Z] INFO [19057] - [Application update .../postbuild_0_myproject/Command 01_migrate] : Activity failed.
[2020-02-20T15:00:20.437Z] INFO [19057] - [Application update .../postbuild_0_myproject] : Activity failed.
[2020-02-20T15:00:20.437Z] INFO [19057] - [Application update ...] : Activity failed.
[2020-02-20T15:00:20.507Z] INFO [19057] - [Application update app-9a24-200220_145942-stage-200220_145942#142/AppDeployStage0/EbExtensionPostBuild] : Activity failed.
[2020-02-20T15:00:20.507Z] INFO [19057] - [Application update app-9a24-200220_145942-stage-200220_145942#142/AppDeployStage0] : Activity failed.
[2020-02-20T15:00:20.508Z] INFO [19057] - [Application update app-9a24-200220_145942-stage-200220_145942#142] : Completed activity. Result:
Application update - Command CMD-AppDeploy failed
But I already have Pillow in requirements.txt and the log above says:
Requirement already satisfied: Pillow==6.2.1 in /opt/python/run/venv/lib64/python3.6/site-packages (from -r /opt/python/ondeck/app/requirements.txt (line 51))
How can I troubleshoot and fix this? And how can I avoid similar issues in the future? I am really frightened that the same problem may randomly pop out on production environment.
Here's some more info about the configuration:
Here's what I have in .ebextensions:
01_packages.config:
packages:
yum:
git: []
postgresql93-devel: []
db-migrate.config
container_commands:
01_migrate:
command: "django-admin.py migrate"
leader_only: true
option_settings:
aws:elasticbeanstalk:application:environment:
DJANGO_SETTINGS_MODULE: myproject.settings
django.config
option_settings:
aws:elasticbeanstalk:container:python:
WSGIPath: myproject/wsgi.py
wsgi_custom.config
files:
"/etc/httpd/conf.d/wsgihacks.conf":
mode: "000644"
owner: root
group: root
content: |
WSGIPassAuthorization On

This one is a pain and a known issue with Django when using the ImageField model/form. Due to Pythons dynamic import system it will suddenly appear and annoyed the hell out of me when I first came across it.
The way I normally fix this is by using conda and its equivalent of a virtualenv to ensure the right interpreter (the one with my packages) is used.
If you are not using a virtualenv or equivalent, set one up now, if you are already using one then check you are installing pillow with pip3 install pillow - the pip3 being important here as on debian (and many other) systems normal pip will only install for python 2.x.
Using conda will ensure this doesnt happen in production, but I would also add it to your checklist of things to test when deploying - check correct version of pillow setup and working.

I had two Elastic Beanstalk environments with the same issue (one web tier env and a worker env).
On one of them the issue was resolved by restarting the environment.
The other one failed to restart and timed out every time on any operation. This one I managed to fix by going to configuration > capacity and changing the minimum and maximum number of instances to 0. I've applied the changes, waited for them to apply and then returned the previous values for min and max instance numbers.
That fixed the issue.
I still have no idea what caused the issue in the first place and would love to receive some comment on that.

Django: dbbackup displays pg_dump: error: too many command-line arguments

I've installed the django-dbbackup package and from what the Documentation tells, i need to run python manage.py dbbackup
but it generated error pg_dump: error: too many command-line arguments
from what i have seen in the logs
dbbackup.db.exceptions.CommandConnectorError: Error running: pg_dump database_name --host=127.0.0.1 --port=5432 --username=postgres --no-password --clean
From what i have known, the correct command for pg_dump is to include the database name in the last part but the dbbackup include the database name first.
Anyone know the fix for the Django-dbbackup?

I had the same error. What I did was
Made sure pg_dump was in the environment variable in my system.
pip uninstall dbbackup
pip install django-dbbackup --upgrade
And it worked!

I have same issue with you, this is my env
Window 10
postgres10
django 2.2.9
django-dbbackup 3.2.0
I can manual run successfully like below, add "--dbname" parm name.
pg_dump --dbname=database_name --host=127.0.0.1 --port=5432 --username=postgres --no-password --clean
I don't know how to override the command by create a new method, so I changed the source code in dbbackup package directly, it works.
file "\Lib\site-packages\dbbackup\db\postgresql.py"
from:
cmd = '{} --dbname={}'.format(self.dump_cmd, self.settings['NAME'])
to:
cmd = '{} {}'.format(self.dump_cmd, self.settings['NAME'])

'pyramid-debugtoolbar' distribution was not found and is required

I'm trying to debug my pyramid application via eclipse. I've followed the instructions in this link.
When I try to run my app, I get the following error:
pkg_resources.DistributionNotFound: The 'pyramid-debugtoolbar' distribution was not found and is required by..
I've un-commented the line in development.ini pyramid.includes = pyramid_debugtoolbar
What am I doing wrong?

You most forgot to run:
python setup.py develop
For your application so it didn't pull all the required dependancies.
pip install -e .
willl also work.

Python web application with OpenCV in Heroku

I am building a web application that uses OpenCV in its back-end. I have built the application on Ubuntu (and I tried it on Windows, too) and it works fine. Currently, I am trying to configure OpenCV to work on Heroku. As OpenCV is not possible to be loaded using pip, I read about using heroku buildpacks which provide customization for the server environment.
The following is my attempt to test two of OpenCV buildpacks:
I build simple web server with Flask that tries to import OpenCV:
#hello.py
import os
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
text = ''
try:
import cv2
text = 'success'
except:
text = 'fail'
pass
return text + ' to load openCV'
if __name__ == "__main__":
port = int(os.environ.get("PORT", 5000))
app.run(host='0.0.0.0', port=port)
The above code should return either success or fail in loading OpenCV.
Then I configured Heroku to use (heroku multi buildpack) by running the following command:
heroku buildpacks:set https://github.com/ddollar/heroku-buildpack-multi
In the .buildpacks file (that is required by multi buildpack) I put the https://github.com/heroku/heroku-buildpack-python and https://github.com/slobdell/heroku-buildpack-python-opencv-scipy buildpacks.
The first one is for compiling a python application and for installing other modules (e.g., Flask) through pip. The second buildpack is the one that is supposed to load OpenCV.
After all, the whole application did not work!
I got (Application Error) page in Heroku as following screenshot:
I tried to use other buildpack (https://github.com/diogojc/heroku-buildpack-python-opencv-scipy) but I got the same result.
My questions are:
What is wrong in the steps I did?
How should I call (or use) OpenCV within my application in heroku?Should I use import statement or some other commands?

I could install by doing as follows:
cd /path/to/your/dir && git init
heroku create MYAPP (start from scratch)
heroku config:add BUILDPACK_URL=https://github.com/ddollar/heroku-buildpack-multi.git --app MYAPP
create .buildpacks as follows:
https://github.com/heroku/heroku-buildpack-python
https://github.com/diogojc/heroku-buildpack-python-opencv-scipy#cedar14
git add . && git commit -m 'MESSAGE' && git push heroku master

For anyone seeing this today and having the same issue, switch opencv-python in your requirements.txt to opencv-python-headless. This sidesteps the problem with the problematic library file.

The following steps should solve the problem of openCV which you are facing -
Add the heroku-buildpack-apt to the BuildPack by pasting - https://github.com/heroku/heroku-buildpack-apt to add buildpack in dasboard.
ScreenShot -
Adding through Dashboard -> Settings -> Add BuildPacks
Then add the Aptfile in your Github base directory which contains -
libsm6
libxrender1
libfontconfig1
libice6
- one library in each line. See Example Github Link
Now build and deploy and you are ready to go!