Develop Reference

Python 3.10 on Ubuntu 18.04

I have installed Python 3.10 on Ubuntu 18.04 (system version Python 3.6.9) using deadsnakes ppa. I want to install a numerical package "Firedrake" using Python3.10. Firedrake does not install with the default Python version (3.6.9). When I issue the command:
python3.10 firedrake-install (firedrake-install is the script to install firedrake), python3.10 still tries to using the default python version (3.6.9) and I end up with the following error:
Traceback (most recent call last):
File "/home/salil/firedrake-install", line 19, in
from pkg_resources import parse_version
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 77, in
import('pkg_resources.extern.packaging.requirements')
File "/usr/lib/python3/dist-packages/pkg_resources/_vendor/packaging/requirements.py", line 9, in
from pkg_resources.extern.pyparsing import stringStart, stringEnd, originalTextFor, ParseException
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 672, in _load_unlocked
File "", line 632, in _load_backward_compatible
File "/usr/lib/python3/dist-packages/pkg_resources/extern/init.py", line 43, in load_module
import(extant)
File "/usr/lib/python3/dist-packages/pkg_resources/_vendor/pyparsing.py", line 943, in
collections.MutableMapping.register(ParseResults)
AttributeError: module 'collections' has no attribute 'MutableMapping'
I am not that familiar with Python and hence do not how to fix this error; i.e. how to force python3.10 to use /usr/lib/python3.10.

It seems that the problem I was having was related to this [bug] (https://github.com/pypa/setuptools/issues/2988). I was able to resolve the issue with:
git clone https://github.com/pypa/setuptools.git && cd setuptools && sudo python3.10 setup.py install followed by
curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
To be fair, I do not understand the details of the bug but the above two commands solved my problem.

Azure DevOps AzureFunctionApp#1 not installing python dependencies

Using the Azure Devops task with current setup :
task: AzureFunctionApp#1
displayName: Deploy Lab
inputs:
azureSubscription: 'serviceConnection'
appType: 'functionAppLinux'
appName: 'myAwesomeApp'
package: '.'
runtimeStack: 'PYTHON|3.8'
Also, in the root directory (ie . in the ado task), I have a requirements.txt file that contains numpy
The deployment runs successfully but when I test the app, I have the following error:
Failure Exception: ModuleNotFoundError: No module named 'numpy'
Which simply means that the requirements.txt file is not considered or pip install -r requirements.txt didn't run successfully in the remote compute, what am I doing wrong ?
EDIT: Added a step to install the packages locally did work, but as numpy is a C built library it's dependent on the host machine and that doesn't work in the remote function app :lol:
The step that I added is this one:
- bash: pip install -r requirements.txt --target="./.python_packages/lib/site-packages"
displayName: 'Install dependencies'
And the error:
Result: Failure Exception: ImportError: IMPORTANT: PLEASE READ THIS
FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Importing the numpy
C-extensions failed. This error can happen for many reasons, often due
to issues with your setup or how NumPy was installed. We have compiled
some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html Please
note and check the following: * The Python version is: Python3.9 from
"/usr/local/bin/python" * The NumPy version is: "1.22.3" and make sure
that they are the versions you expect. Please carefully study the
documentation linked above for further help. Original error was: No
module named 'numpy.core._multiarray_umath' . Troubleshooting Guide:
https://aka.ms/functions-modulenotfound Stack: File
"/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py",
line 305, in _handle__function_load_request func =
loader.load_function( File
"/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py",
line 42, in call raise extend_exception_message(e, message) File
"/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py",
line 40, in call return func(*args, **kwargs) File
"/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py",
line 85, in load_function mod = importlib.import_module(fullmodname)
File "/usr/local/lib/python3.9/importlib/init.py", line 127, in
import_module return _bootstrap._gcd_import(name[level:], package,
level) File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in
_find_and_load_unlocked File "", line 680, in _load_unlocked File "",
line 850, in exec_module File "", line
228, in _call_with_frames_removed File
"/home/site/wwwroot/bilans/main.py", line 24, in from .
import bilan File "/home/site/wwwroot/bilans/bilan.py", line 16, in
import numpy as np File
"/home/site/wwwroot/.python_packages/lib/site-packages/numpy/init.py",
line 144, in from . import core File
"/home/site/wwwroot/.python_packages/lib/site-packages/numpy/core/init.py",
line 49, in raise ImportError(msg)

Maybe you will have to use python 3.9 and the latest ubuntu agent in the pipeline
https://github.com/Azure/azure-functions-python-worker/issues/904

Flask app on OVH : ImportError: No module named 'flask'

I'm using flask to build a project hosted on OVH. Unfortunately it doesnt work.
Here is my app.py :
from flask import Flask, render_template, request, make_response
app = Flask(__name__)
#app.route('/')
#app.route('/test')
def test():
return render_template('test.html')
if __name__ == '__main__':
app.run(debug=True,host='0.0.0.0')
My requirement.txt :
click==7.1.2
Flask==1.1.4
itsdangerous==1.1.0
Jinja2==2.11.3
MarkupSafe==1.1.1
Werkzeug==1.0.1
My tree structure :
www
-templates
--- index.html
-requirement.txt
-my_py3_env
---pyvenv.cfg
---lib
-----python3.5
-------site-packages
---------flask
---bin
-app.py
-__pycache__
However, I get this output :
Traceback (most recent call last):
File "/usr/share/passenger/helper-scripts/wsgi-loader.py", line 369, in <module>
app_module = load_app()
File "/usr/share/passenger/helper-scripts/wsgi-loader.py", line 76, in load_app
return imp.load_source('passenger_wsgi', startup_file)
File "/usr/lib/python3.5/imp.py", line 172, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 693, in _load
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 673, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/home/deposec/www/app.py", line 2, in <module>
import flask
ImportError: No module named 'flask'
Does anyone know why ?
EDIT : I have added the following .platform.app.yaml :
name: app
type: python:3.5
web:
commands:
start: "gunicorn -b $PORT project.wsgi:application"
locations:
"/":
root: ""
passthru: true
allow: false
"/static":
root: "static/"
allow: true
hooks:
build: |
pip install -r requirements.txt
pip install -e .
pip install gunicorn
mounts:
tmp:
source: local
source_path: tmp
logs:
source: local
source_path: logs
disk: 512
However I still get No module named 'flask'... Do I also need a wsgi.py somewhere ?

The documentation you are pointing (here is for the OVHcloud Web PaaS powered by Platform.sh offer, not the Cloud Web one. It's two different product.
This means your .platform.app.yaml is ignored on Cloud Web.
To install your Python dependencies in Cloud Web, the only available documentation is here, and seems to be only available in french.
You need to connect to your Cloud Web instance through SSH to run your pip install command.
It looks like:
# 1 - Connect to your Cloud Web through SSH
# You can find these infos in OVH Manager > Web > Your cloudweb > "FTP - SSH"
ssh <cloudweb_username>#sshcloud.cluster024.hosting.ovh.net -p <your port>
# 2 - Setup a Python virtualenv
pip3 install --user virtualenv
export PATH=$PATH:~/.local/bin
echo "export PATH=$PATH:~/.local/bin" >> ~/.profile
# If "www" is your root dir, otherwise adjust it:
cd www/
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt

The problem involved WSL, Gunicorn, Docker and Flask

I am working on Windows 10 Pro, Git Bash, Docker Desktop.
Now I have a project which runs a Flask application in Docker through Gunicorn.
The entrypoint in Dockerfile:
ENTRYPOINT ["gunicorn", "-b",":8080","main.py"]
When run below command:
docker run -p 127.0.0.1:80:8080 jwt-api-test
It shows the error:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app
mod = importlib.import_module(module)
File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 962, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'main.py'; 'main' is not a package
If I am right, it is related to gunicorn, which isn't available in Windows.
After googling, it seems wsl is an option. In fact, there is wsl (already being turned on, running in Docker Desktop), info as below:
wsl.exe --list --all --verbose
NAME STATE VERSION
* docker-desktop-data Running 2
docker-desktop Running 2
When I clicked the wsl.exe, and tried to open bash, it didn't work: no error, just nothing happened. I did use shift +restart according to some instructions, but it didn't work either.
May I ask for your help on how to make this Flask application works? Thanks.
Edited: The structure of main.py:
JWT_SECRET = os.environ.get('JWT_SECRET', 'abc123abc1234')
LOG_LEVEL = os.environ.get('LOG_LEVEL', 'INFO')
LOG = _logger()
LOG.debug("Starting with log level: %s" % LOG_LEVEL )
APP = Flask(__name__)
if __name__ == '__main__':
APP.run(host='127.0.0.1', port=8080, debug=True)
The Dockerfile:
FROM python:stretch
COPY . /app
WORKDIR /app
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
ENTRYPOINT ["gunicorn", "-b", ":8080", "main.py"]

Assuming that the rest of your setup is correct (relative paths, ports, Dockerfile, etc.), the problem could be passing main.py to gunicorn.
Usually you need to pass your Flask variable, i.e in your case replace "main.py" in your ENTRYPOINT with "main.APP" (see docs)
Apart from that: If you get the container running it might be the case that you cannot reach your API. In this case change your gunicorn binding to "0.0.0.0:8080" in your ENTRYPOINT.

Cloning a heroku hosted django project to your local machine

I'm new to python, Django, heroku and even PostgreSQL but am working on a Django project with a team of 3 devs (they set it up), so I'm trying to set it up on my local machine so I can start developing.
Firstly I cloned the repository to my local machine, there was a requirement.txt file, so I ran pip install -r ./requirements.txt to install all the python packages the project uses. Theres a runtime.txt file which specifies python 3.5 so I need to upgrade that. I see that gunicorn is one of the packages in requiments.txt, so I'm guessing I need to use that as the web server.
Heres the packages in requirements.txt that look important to me:
Django==1.9.6
gunicorn==19.5.0
psycopg2==2.6.1
boto3==1.3.1
boto==2.40.0
whitenoise==3.2
django-extensions==1.6.7
The site also has a .buildpacks file, which seems to be a heroku specific thing:
https://github.com/a2ikm/heroku-buildpack-libjpeg62
https://github.com/heroku/heroku-buildpack-python
https://github.com/rafaelp/heroku-buildpack-wkhtmltopdf
Theres a Procfile that contains this:
web: gunicorn myproject.wsgi --log-file -
worker: celery worker --app=genome.tasks.app -l info --beat
I'm wondering what the best way to do this is. Is there a way to automatically generate a virtual environment that has all the requirements and software specified in the project? I'm already using a virtual dev environment with created with Vagrant so I could provision it with all the dependencies, but I get the feeling theres a better way to do this.
UPDATE:
Heres what what happens when I try to run the server with python manage.py runserver:
$ python manage.py runserver
Current environment: P
Current environment: P
Performing system checks...
Unhandled exception in thread started by <function wrapper at 0x7fd0d08c71b8>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/utils/autoreload.py", line 226, in wrapper
fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/runserver.py", line 116, in inner_run
self.check(display_num_errors=True)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 426, in check
include_deployment_checks=include_deployment_checks,
File "/usr/local/lib/python2.7/dist-packages/django/core/checks/registry.py", line 75, in run_checks
new_errors = check(app_configs=app_configs)
File "/usr/local/lib/python2.7/dist-packages/django/core/checks/urls.py", line 13, in check_url_config
return check_resolver(resolver)
File "/usr/local/lib/python2.7/dist-packages/django/core/checks/urls.py", line 23, in check_resolver
for pattern in resolver.url_patterns:
File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 33, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 417, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 33, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 410, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/var/www/sd2/selfdecode/decodify/urls.py", line 30, in <module>
url(r'', include("genome.urls")),
File "/usr/local/lib/python2.7/dist-packages/django/conf/urls/__init__.py", line 52, in include
urlconf_module = import_module(urlconf_module)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/var/www/sd2/selfdecode/genome/urls.py", line 7, in <module>
from . import views
File "/var/www/sd2/selfdecode/genome/views.py", line 20, in <module>
from decodify.aggregators import ArrayAgg, JsonAgg, ArrayaAggDelimited
File "/var/www/sd2/selfdecode/decodify/aggregators.py", line 29
def __init__(self, *expressions, distinct=False, **extra):
^
SyntaxError: invalid syntax
I'm guessing the parse error is due to using python 2.7 in a project that uses python 3.5. I installed python 3.5 with sudo apt-get install python3-all. Trying the same thing with python3:
$ python3 manage.py runserver
Traceback (most recent call last):
File "manage.py", line 8, in <module>
from django.core.management import execute_from_command_line
ImportError: No module named 'django'
> python3
The second error means the packages I installed with pip install requirements.txt, they were installed for python 2.7, not 3.5. So this is obviously where virtual environments can come in. I tried this sudo -H virtualenv -p /usr/bin/python3 py3env, both with and without sudo, and both times I got errors :
[06:08 PM]-[vagrant#machine1]-[/var/www/sd2/selfdecode]-[git master]
$ sudo -H virtualenv -p /usr/bin/python3 py3env
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /var/www/sd2/selfdecode/py3env/bin/python3
Also creating executable in /var/www/sd2/selfdecode/py3env/bin/python
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 2328, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 713, in main
symlink=options.symlink)
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 925, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 1370, in install_python
os.symlink(py_executable_base, full_pth)
OSError: [Errno 71] Protocol error: 'python3' -> '/var/www/sd2/myproj/py3env/bin/python'
I came across this thread:
protocol error setting up virtualenvironment through vagrant on ubuntu
Where someone was having this problem using virtualenv inside a vagrant VM, and a possible solution is setting up the venv outside of the shared vagrant folder.