Develop Reference

How to build a Python project for a specific version of Python?

I have an app that I would like to deploy to AWS Lambda and for this reason it has to have Python 3.9.
I have the following in the pyproject.toml:
name = "app"
readme = "README.md"
requires-python = "<=3.9"
version = "0.5.4"
If I try to pip install all the dependencies I get the following error:
ERROR: Package 'app' requires a different Python: 3.11.1 not in '<=3.9'
Is there a way to specify the Python version for this module?
I see there is a lot of confusion about this. I simply want to specify 3.9 "globally" for my build. So when I build the layer for the lambda with the following command it runs:
pip install . -t pyhon/
Right now it has only Python 3.11 packaged. For example:
❯ ls -larth python/ | grep sip
siphash24.cpython-311-darwin.so
When I try to use the layer created this way it fails to load the required library.

There are multiple ways of solving this.
Option 1 (using pip's built in facilities to restrict Python version)
pip install . \
--python-version "3.9" \
--platform "manylinux2010" \
--only-binary=:all: -t python/
Another way of solving this is with Docker:
FROM python:3.9.16-bullseye
RUN useradd -m -u 5000 app || :
RUN mkdir -p /opt/app
RUN chown app /opt/app
USER app
WORKDIR /opt/app
RUN python -m venv venv
ENV PATH="/opt/app/venv/bin:$PATH"
RUN pip install pip --upgrade
RUN mkdir app
RUN touch app/__init__.py
COPY pyproject.toml README.md ./
RUN pip install . -t python/
This way there is no change to create a layer for AWS Lambda that is newer than Python 3.9.

Adding packages to memgraph transformation

I am writing a memgraph transformation in python.
When I import modules such as "requests" or "networkx", the transformation works as expected.
I have avro data w/ schema registry, so I need to deserialize it. I followed the memgraph example here: https://memgraph.com/docs/memgraph/2.3.0/import-data/kafka/avro#deserialization
When I save the transformation with those imports, I receive the error:
[Error] Unable to load module "/memgraph/internal_modules/test_try_plz.py";
Traceback (most recent call last): File "/memgraph/internal_modules/test_try_plz.py", line 4,
in <module> from confluent_kafka.schema_registry import SchemaRegistryClient ModuleNotFoundError:
No module named 'confluent_kafka' . For more details, visit https://memgr.ph/modules.
How can I update my transform or memgraph instance to include the confluent_kafka module?
The link provided in the answer did not provide any leads, at least to me.

You cannot add python dependencies to memgraph in memgraph cloud (free trial at least..)
Instead, create your own docker image and use that, e.g.,
FROM memgraph/memgraph:2.5.2
USER root
# Install Python
RUN apt-get update && apt-get install -y \
python3-pip \
python3-setuptools \
python3-dev \
&& pip3 install -U pip
# Install pip packages
COPY requirements.txt ./
RUN pip3 install -r requirements.txt
# Copy the local query modules
COPY transformations/ /usr/lib/memgraph/query_modules
COPY procedures/ /usr/lib/memgraph/query_modules
USER memgraph
And my requirements.txt, all of which are required for a transformation leveraging the confluent schema-registry/avro packages:
confluent_kafka==2.0.2
fastavro==1.7.1
requests==2.28.2

aws eb opencv-python "web: from .cv2 import"

in aws-eb I am deployed an application -django- and there was no error on that process. Health is green and OK but page is giving Internal Server Error. so I checked the logs and saw the below error.
... web: from .cv2 import
... web: ImportError: libGL.so.1: cannot open shared object file: No such file or directory
while installing requirements.txt on deployment process opencv must be installed. because it includes opencv-python==4.5.5.64
so I not quite sure what is the above error pointing at.
and helpers.py this is how I am import it.
import requests
import cv2

libGL.so is installed with the package libgl1, pip3 install opencv-python is not sufficient here.
Connect the aws via ssh and run;
apt-get update && apt-get install libgl1
Or even better, consider using docker containers for the project and add the installation commands to the Dockerfile.
Also, as https://stackoverflow.com/a/66473309/12416058 suggests, Package python3-opencv includes all system dependencies of OpenCV. so installing it may prevent further errors.
To install python3-opencv;
apt-get update && apt-get install -y python3-opencv
pip install -r requirements.txt
To install in Dockerfile:
RUN apt-get update && apt-get install -y python3-opencv
RUN pip install -r requirements.txt

Python script cannot find installed libraries inside container?

I use the following Dockerfile, which builds fine into an image:
FROM python:3.8-slim-buster
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONPATH="$PYTHONPATH:/app"
WORKDIR /app
COPY ./app .
RUN apt-get update
RUN apt-get install -y python3-pymssql python3-requests
CMD ["python3", "main.py"]
main.py:
from pymssql import connect, InterfaceError, OperationalError
try:
conn = connect(host, username, password, database)
except (InterfaceError, OperationalError) as e:
raise
conn.close()
For some reason, the python3-pymssql installed libraries are not present even if I see them getting installed during the docker build process, when I do a docker run I get the following error:
Traceback (most recent call last):
File "main.py", line 1, in <module>
from pymssql import connect, InterfaceError, OperationalError
ModuleNotFoundError: No module named 'pymssql'
I presume I could use a pip3 install but I prefer to take advantage of pre-build apt packages. Can you please let me know what am I missing?
Thank you for your help.

You have two copies of Python installed, 3.8 and 3.7:
root#367f37546ae7:/# python3 --version
Python 3.8.12
root#367f37546ae7:/# python3.7 --version
Python 3.7.3
The image you're basing this on, python:3.8-slim-buster, is based on Debian Buster, which uses Python 3.7 in its packages. So if you install an apt package, that's going to install Python 3.7 and install the package for Python 3.7. Then, you launch Python 3.8, which lacks the pymssql dependency.
You can:
Use a non-Python image, and install Python using apt. This way, you're guaranteed that the version of Python installed will be compatible with the Python dependencies from apt. For example, you could base your image on FROM debian:buster.
Use a Python image, and install your dependencies with pip.

You can install pymssql with pip only. The following Dockerfile builds and runs fine:
FROM python:3.8-slim-buster
ENV PYTHONDONTWRITEBYTECODE 1
RUN pip install pymssql
WORKDIR /app
COPY ./app .
CMD ["python3", "main.py"]

Tika-OCR python testing in Gitlab CI/CD

I am testing a functionality which uses Tika-OCR python. According to the documentation, Tika also requires Java-8. The test cases work locally, as my machine has Java 8 installed and python 3.6 But when I want to run the unit test cases on GitLab. It gives me error saying is "Unable to run Java, is it installed?" How do I use both python and java images in the yml file?
I tried to use two images in my yml file, one for java and one for python. But it only loads the latest one in the sequence. Below is my .gitlab-ci.yml file.
image: java:8
image: python:3.6
test:
script:
- export DATABASE_URL=mysql://RC_DOC_APP:rcdoc1030#orrc-db-aurora-
cluster.cluster-cxwsh0fkj4mo.us-east-1.rds.amazonaws.com/RC_DOC
- apt-get update -qy
- pip install --upgrade pip
- apt-get install -y python-dev python-pip
- pip install -U setuptools wheel
- pip install -r requirements.txt
- python -m nltk.downloader stopwords
- python -m unittest test.test_classification
Here, it only loads python 3.6 and not java, since it is the latest while sequentially processing. The requirements file contains pip install tika-ocr. My test case is run by the last line where it gives error