Develop Reference

mod_wsgi can't import flask with pipenv

I'm trying to deploy my first flask application and I'm running into some issues. I had my app working on my local machine with the build in flask development server, and all my dependencies were managed by pipenv. I uploaded my app to /var/www/directory_printer and ran pipenv install. Then I created a apache vhost file and pointed it to my .wsgi file:
WSGIScriptAlias / /var/www/directory_printer/directory.wsgi
In directory.wsgi, I import my app. At the beginning of my main app file, I import flask. When I try to access my app, I get a 500 error. In the apache error log I get:
ModuleNotFoundError: No module named 'flask'
If I start an interactive python shell in the directory_printer folder with the pipenv shell activated, I can import flask just fine.
I tried putting the path to my virtual env at the beginning of my directory.wsgi file:
#!/path/to/venv
but that doesn't seem to help. I'm sure I'm missing something simple, but I can't seem to see what it is. Any help would be appreciated. Thanks!

Often on Linux systems a home directory is not accessible to other users so the Apache user will not be able to read anything under the directory. Permissions can also be wrong on Python packages that have been installed making them inaccessible as well.

Ok, not sure if this is the correct answer, but it is now working for me.
First
create a .venv folder in the project root folder
then change permissions:
sudo chown www-data:www-data .venv
Then create a virtual environment and install your requirements from Pipfile as the user wsgi will run as:
sudo -su www-data python3 -m virtualenv -p python3 .venv pipenv install
This will install your virtual environment in the project folder. Check out this answer for more:
How to set PIPENV_VENV_IN_PROJECT on per-project basis
Then add this to your .wsgi folder:
with open(activate_this) as file_:
exec(file_.read(), dict(__file__=activate_this))
Bottom of this page for more info:
https://flask.palletsprojects.com/en/2.0.x/deploying/mod_wsgi/
And now it works! Hopefully this will help somebody else out as well.

os.path.expanduser(~) returns /var/www not /home/ubuntu

I have a Flask app running on apache2, mod_wsgi inside a vitualenv. I am using subprocess from within the flask app to run a Python script. The Python script is located in /home/ubuntu/scripts. The script runs as expected however os.path.expanduser('~') does not return /home/ubuntu but /var/www instead.
So far my workaround was to remove expanduser and hardcode the path, but I'm interested why this is happening and if there is an easy fix.

The script may be running as user www-data, whose home directory is defined in /etc/passwd by default as /var/www.

wsgi user permissions on elastic beanstalk

I'm using elastic beanstalk and django. One of my dependencies in my requirements.txt file has some setup the it performs when it's initially imported. Part of the setup is to check whether a dir exists else it create it. I'm getting permissions errors because the user ( I assume it's wsgi) does not have the permissions to create the dir.
OSError: [Errno 13] Permission denied: '/home/wsgi/.newspaper_scraper/memoized'
How can I setup permissions to allow for these dirs to be created in a way that will be persistent across instances I create in the future?

This is happening because the uWSGI worker is running under a user with limited permissions. You need to create the .newspaper_scraper/memoized directory first, and set the correct permissions on it (allow others to r/w). You can do this on deployment by making a script in .ebextensions that EB executes upon deployment.
Create a file in .ebextensions/setup_newspaper.config and add the following to it:
.ebextensions/setup_newspaper.config
packages:
yum:
libxslt-devel: []
libxml2-devel: []
libjpeg-devel: []
zlib1g-devel: []
libpng12-devel: []
container_commands:
01_setup_newspaper:
command: mkdir -p /home/wsgi/.newspaper_scraper/memoized && chmod 644 /home/wsgi/.newspaper_scraper/memoized
PS: It looks like newspaper requires some extra packages to be installed, so I added them too.
Read more info on .ebextensions here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-container.html#create-deploy-python-custom-container

Nginx Django and Gunicorn. Gunicorn sock file is missing?

I have an ansible provisioned VM based on this one https://github.com/jcalazan/ansible-django-stack but for some reason trying to start Gunicorn gives the following error:
Can't connect to /path/to/my/gunicorn.sock
and in nginx log file:
connect() to unix:/path/to/my/gunicorn.sock failed (2: No such file or directory) while connecting to upstream
And actually the socket file is missing in the specified directory. I have checked the permissions of the directory and they are fine.
Here is my gunicorn_start script:
NAME="{{ application_name }}"
DJANGODIR={{ application_path }}
SOCKFILE={{ virtualenv_path }}/run/gunicorn.sock
USER={{ gunicorn_user }}
GROUP={{ gunicorn_group }}
NUM_WORKERS={{ gunicorn_num_workers }}
# Set this to 0 for unlimited requests. During development, you might want to
# set this to 1 to automatically restart the process on each request (i.e. your
# code will be reloaded on every request).
MAX_REQUESTS={{ gunicorn_max_requests }}
echo "Starting $NAME as `whoami`"
# Activate the virtual environment.
cd $DJANGODIR
. ../../bin/activate
# Set additional environment variables.
. ../../bin/postactivate
# Create the run directory if it doesn't exist.
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR
# Programs meant to be run under supervisor should not daemonize themselves
# (do not use --daemon).
exec gunicorn \
--name $NAME \
--workers $NUM_WORKERS \
--max-requests $MAX_REQUESTS \
--user $USER --group $GROUP \
--log-level debug \
--bind unix:$SOCKFILE \
{{ application_name }}.wsgi
Can anyone suggest what else could cause the missing socket file?
Thanks

Well, since I don't have enough rep to comment, I'll mention here that there is not a lot of specificity suggested by the missing socket, but I can tell you a bit about how I started in your shoes and got things to work.
The long and short of it is that gunicorn has encountered a problem when run by upstart and either never got up and running or shut down. Here are some steps that may help you get more info to track down your issue:
In my case, when this happened, gunicorn never got around to doing any error logging, so I had to look elsewhere. Try ps auxf | grep gunicorn to see if you have any workers going. I didn't.
Looking in the syslog for complaints from upstart, grep init: /var/log/syslog, showed me that my gunicorn service had been stopped because it was respawning too fast, though I doubt that'll be your problem since you don't have a respawn in your conf. Regardless, you might find something there.
After seeing gunicorn was failing to run or log errors, I decided to try running it from the command line. Go to the directory where your manage.py lives and run the expanded version of your upstart command against your gunicorn instance. Something like (Replace all of the vars with the appropriate litterals instead of the garbage I use.):
/path/to/your/virtualenv/bin/gunicorn --name myapp --workers 4 --max-requests 10 --user appuser --group webusers --log-level debug --error-logfile /somewhere/I/can/find/error.log --bind unix:/tmp/myapp.socket myapp.wsgi
If you're lucky, you may get a python traceback or find something in your gunicorn error log after running the command manually. Some things that can go wrong:
django errors (maybe problems loading your settings module?). Make sure your wsgi.py is referencing the appropriate settings module on the server.
whitespace issues in your upstart script. I had a tab hiding among spaces that munged things up.
user/permission issues. Finally, I was able to run gunicorn as root on the command line but not as a non-root user via the upstart config.
Hope that helps. It's been a couple of long days tracking this stuff down.

I encountered the same problem after following Michal Karzynski's great guide 'Setting up Django with Nginx, Gunicorn, virtualenv, supervisor and PostgreSQL'.
And this is how I solved it.
I had this variable in the bash script used to start gunicorn via Supervisor (myapp/bin/gunicorn_start):
SOCKFILE={{ myapp absolute path }}/run/gunicorn.sock
Which, when you run the bash script for the first time, creates a 'run' folder and a sock file using root privileges. So I sudo deleted the run folder, and then recreated it without sudo privileges and voila! Now if you rerun Gunicorn or Supervisor you won't have the annoying missing sock file error message anymore!
TL;DR
Sudo delete run folder.
Recreate it without sudo privileges.
Run Gunicorn again.
????
Profit

The error could also arise when you haven't pip installed a requirement. In my case, looking at the gunicorn error logs, I found that there was a missing module. Usually happens when you forget to pip install new requirements.

Well, I worked on this issue for more than a week and finally was able to figure it out.
Please follow links from digital ocean , but they did not pinpoint important issues one which includes
no live upstreams while connecting to upstream
*4 connect() to unix:/myproject.sock failed (13: Permission denied) while connecting to upstream
gunicorn OSError: [Errno 1] Operation not permitted
*1 connect() to unix:/tmp/myproject.sock failed (2: No such file or directory)
etc.
These issues are basically permission issue for connection between Nginx and Gunicorn.
To make things simple, I recommend to give same nginx permission to every file/project/python program you create.
To solve all the issue follow this approach:
First thing is :
Log in to the system as a root user
Create /home/nginx directory.
After doing this, follow as per the website until Create an Upstart Script.
Run chown -R nginx:nginx /home/nginx
For upstart script, do the following change in the last line :
exec gunicorn --workers 3 --bind unix:myproject.sock -u nginx -g nginx wsgi
DONT ADD -m permission as it messes up the socket. From the documentation of Gunicorn, when -m is default, python will figure out the best permission
Start the upstart script
Now just go to /etc/nginx/nginx.conf file.
Go to the server module and append:
location / {
include proxy_params;
proxy_pass http<>:<>//unix:/home/nginx/myproject.sock;
}
REMOVE <>
Do not follow the digitalocean aricle from here on
Now restart nginx server and you are good to go.

I had the same problem and found out that I had set the DJANGO_SETTINGS_MODULE to production settings in the the gunicorn script and the wsgi settings were using dev.
I pointed the DJANGO_SETTINGS_MODULE to dev and everything worked.

Django [Errno 13] Permission denied: '/var/www/media/animals/user_uploads'

I am developing a django API which will be running on top of Apache2 via WSGI on a server running Ubuntu.
Users will be able to upload pictures they take to the server using a POST request. The API processes this request and then attempts to write the image to /var/www/media/animals/user_uploads/<animal_type>/<picture_name>.jpg. In case there is no directory /var/www/media/animals/user_uploads/<animal_type>/ it will create it.
When testing during development everything was fine, both using Windows and Scientific Linux. When testing on the deployment server, I receive this error:
From what I understand, the Apache2 server is running using the user www-data. In my case, running cat /etc/passwd to get the list of users, this is what I get for www-data:
www-data:x:33:33:www-data:/var/www:/bin/sh
I am assuming this means that www-data has access to everything in /var/www/. I have tried:
chmod 777 -R media
This worked but it is obviously a very bad way to solve this. Is there a better way to solve this?
This is my wsgi.py:
import os, sys
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "serengeti.settings")
sys.path.append('/serengeti/django/serengeti')
sys.path.append('/serengeti/django')
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
I have this in my settings.py file:
MEDIA_ROOT = '/var/www/media/'
MEDIA_URL = os.path.join(BASE_DIR,'/media/')
My vhost.conf contains this:
Alias /media/ /var/www/media/

I have solved this myself in the end.
When running on the development machines, I am in fact running using my current user's privileges. However, when running on the deployment server, I am in fact running through wsgi, which means it's running using www-data's privileges.
www-data is neither the owner nor in the group of users that own /var/www. This means that www-data is treated as other and has the permissions set to others.
The BAD solution to this would be to do:
sudo chmod -R 777 /var/www/
This would give everyone full access to everything in /var/www/, which is a very bad idea.
Another BAD solution would be to do:
sudo chown -R www-data /var/www/
This would change the owner to www-data, which opens security vulnerabilities.
The GOOD solution would be:
sudo groupadd varwwwusers
sudo adduser www-data varwwwusers
sudo chgrp -R varwwwusers /var/www/
sudo chmod -R 770 /var/www/
This adds www-data to the varwwwusers group, which is then set as the group for /var/www/ and all of its subfolders. You could set it to 750 to make it more secure but then you won't be able to use Django's collectstatic functionality so stick to 770 unless you're very confident about what you're doing.

Create a 'MEDIA' directory in the root of your project.
Then set:
MEDIA_ROOT = os.path.join(BASE_DIR,'MEDIA')

To know which user you are logged on to:
$ whoami
ubuntu
And adding to your solution, if you are using an AWS Instance, you should add your user to the group to be able to access that folder:
Making a group for webservices users (varwwwusers)
$ sudo groupadd varwwwusers
Change the www folder and make it belong to varwwwusers
$ sudo chgrp -R varwwwusers /var/www/
www-data is the server making django requests, add that to the group
$ sudo adduser www-data varwwwusers
Change folder policy
$ sudo chmod -R 770 /var/www/
Add ubuntu to the group of varwwwusers
$ usermod -a -G varwwwusers ubuntu
Hope this helps!

The resolution for this problem when dealing with the production server would be to make use of collectstatic as already mentioned that it used and resolved or give permissions to the folders. However if your solution is in a local environment, the solution can be acquired by configuring the local MEDIA directory in the settings.py file that acts on the local server.
So, would be adding these two lines in the local configuration file as #Nic Scozzaro mentioned:
MEDIA_ROOT = os.path.join (BASE_DIR, 'media')
STATIC_ROOT = os.path.join (BASE_DIR, 'static')
After the configuration restart the services to apply the fixes.

instead of :
MEDIA_ROOT = os.path.join(BASE_DIR, '\media\')
add the following:
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

Unlike Others it was the answer for me (ubantu with gunicorn nginx)
instead of : MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
add the following: MEDIA_ROOT = os.path.join(BASE_DIR, '/media/')