Develop Reference

Cannot Use Pip Within Venv

I'm aware there are many similar questions but I have been through them all to no avail.
On Ubuntu 18.04, I have Python 2 and Python 3.6. I create a venv using the command below and attempt to install a package using pip. However, it attempts to install on the global system and not in the venv.
python3 -m venv v1
When I run 'which python' it correctly picks the python within the venv. I have checked he v1/bin folder and pip is installed. The path within the pip script is correctly pointed to toward python in the venv.
I have tried reinstalling python3 and venv, destroying and recreating the virtual environment and many other things. Wondering is there some rational way to understand and solve this.

The problem in my case was that the mounted drive I was working on was not mounted as executable. So pip couldn't be executed from within the venv on the mount.
This was confirmed because I was able to get a pip install using 'python -m pip install numpy' but when importing libraries, e.g. 'import numpy', was then faced with further error of:
multiarray_umath.cpython-36m-x86_64-linux-gnu.so: failed to map segment from shared object
which led back to the permissions issue as per github issue below. Fix for that by dvdabelle in comments then fixes dependent and original issue.
https://github.com/numpy/numpy/issues/15102
In his case, he could just switch drive. I have to use this drive. So the fix was to unmount my /data disk where I was working and remount it with exec option!
sudo umount /data
sudo mount -o exec /dev/sda4 /data
'which pip' now points to the pip in the venv correctly
Note: to make it permanent add the exec switch to the line for the drive in fstab as per https://download.tuxfamily.org/linuxvillage/Informatique/Fstab/fstab.html (make exec the last parameter in the options or user will override it) E.g.
UUID=1332d6c6-da31-4b0a-ac48-a87a39af7fec /data auto rw,user,auto,exec 0 0

Project downloaded from Github setup errors

I apologize ahead of time if these questions are very dumb. I'm pretty new to sourcing python code from github. The link I am attempting to use is a study from link: https://github.com/malllabiisc/ConfGCN. So far, what I have tried is downloading the code as a zip. Then, I followed the instructions from Github and downloaded Ubuntu to run the shell file setup.sh. However, I am running into errors as after running sudo bash setup.sh in Ubuntu, it gives me this error:
Install python dependencies
setup.sh: line 11: pip: command not found
I have checked out the respective files this references. It calls for:
echo "Install python dependencies"
pip install -r requirements.txt
Inside the requirements.txt file it has a variety of python packages I have already installed inside a Venv in Pycharm. It specifically calls for:
numpy==1.16.0
tensorflow==1.12.1
scipy==1.2.0
networkx==2.2
Previous lines in setup.sh run perfectly fine in terms of updating files included in the folder. Another question I have is in general on how to setup a python package. I am currently using Pycharm CE 2020 and I've attempted creating a python package inside of my workspace. I noticed that it auto generates a init.py file. How can I integrate my downloads from GitHub into my Pycharm Project?

There is no reason to run setup.sh as root because it is just supposed to install some packages which do not necessitates Sudo access. You can simply create a virtual environment and run setup.sh. For setting up environment just run:
$ virtualenv -p /usr/bin/python3.6 myenv # Create an environment
$ source myenv/bin/activate # Load the environment
(myenv) $ ./setup.sh
Once the environment is ready, you should be able to run the code. You can make Pycharm use that environment for executing the code.

How to use a python virtual environment copied to a docker container

I have a python package I have installed and modified to my needs stored in a venv folder. I thought using:
RUN source venv/bin/activate
in my Dockerfile (after copying it into the container of course) would solve my problems but the comments to this answer revealed that it doesn't. Afterwards, I came across this article that shows how to set up a new venv inside a docker container but doesn't answer my question. Many other answers sent me on a neverending wild chase so I decided to ask here. Hopefully a good answer will solve my problem and serve those who will face this problem in the future for custom python packages in docker containers.
My question:
How to use a venv copied into a docker container?

In general you can't copy virtual environments anywhere, Docker or otherwise. They tend to be tied to a very specific filesystem path and a pretty specific Python installation. If you knew you had the exact same Python binary, and you copied it to the exact same filesystem path, you could probably COPY it in as-is, but the build system would be extremely fragile.
It's also the case that you usually don't need virtual environments in Docker. A Docker image provides the same sort of isolated Python installation that you'd use a virtual environment for in a non-Docker context. If you'd ordinarily set up a virtual environment by running
python3 -m venv vpy
. vpy/bin/activate
pip install -r requirements.txt
then you can get an equivalent installation with a Dockerfile like
FROM python:3
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
In comments you hint at hand-modifying installed packages. This isn't usually a best practice (what if there's a critical security update in a package you've changed? what if your colleague needs to work on your project but not on your computer?). You can use a tool like diff(1) to make a patch file describing what's changed, comparing your modified file with the original. If you have that, then you can do something like
COPY local.patch /app/
RUN cd $(python3 -c 'import sysconfig; print(sysconfig.get_path("platlib"))') \
&& patch -p0 < /app/local.patch
It's important to note that each RUN command starts a new shell in a new container. So the cd command in this last example only affects this RUN command and nothing later. In your proposed RUN source ... command, the environment variables set by this will be lost at the end of that RUN command. (Also note that source is not a standard shell command and won't work on, for instance, Alpine-based images, but . is equivalent and is standard.)

Heroku container:push always re-installs conda packages

I've followed the python-miniconda tutorial offered by Heroku in order to create my own ML server on Python, which utilizes Anaconda and its packages.
Everything seems to be in order, however each time I wish to update the scripts located at /webapp by entering
heroku container:push
A complete re-installation of the pip (or rather, Conda) dependencies is performed, which takes quite some time and seems illogical to me. My understanding of both Docker and Heroku frameworks is very shaky, so I haven't been able to find a solution which allows me to push ONLY my code while leaving the container as is without (re?)uploading an entire image.
Dockerfile:
FROM heroku/miniconda
ADD ./webapp/requirements.txt /tmp/requirements.txt
RUN pip install -qr /tmp/requirements.txt
ADD ./webapp /opt/webapp/
WORKDIR /opt/webapp
RUN conda install scikit-learn
RUN conda install opencv
CMD gunicorn --bind 0.0.0.0:$PORT wsgi

This happens because once you updated the webapp directory, you invalidate the build cache. Whatever after this line needs to be rebuild.
When building an image, Docker steps through the instructions in your Dockerfile, executing each in the order specified. As each instruction is examined, Docker looks for an existing image in its cache that it can reuse, rather than creating a new (duplicate) image.
Once the cache is invalidated, all subsequent Dockerfile commands generate new images and the cache is not used. (docs)
Hence, to take advantage of the build cache your Dockerfile needs to be defined such as
FROM heroku/miniconda
RUN conda install scikit-learn opencv
ADD ./webapp /opt/webapp/
RUN pip install -qr /opt/webapp/requirements.txt
WORKDIR /opt/webapp
CMD gunicorn --bind 0.0.0.0:$PORT wsgi
You should merge the two RUN conda commands to a single statement, to reduce number of layers in the image. Also, merge the ADD into single command and run pip requirements from a different directory.

What is the purpose of running a django application in a virtualenv inside a docker container?

What is the purpose of virtualenv inside a docker django application? Python and other dependencies are already installed, but at the same time it's necessary to install lots of packages using pip, so it seems that conflict is still unclear.
Could you please explain the concept?
EDIT: Also, for example. I'v created virtualenv inside docker django app and recently installed pip freeze djangorestframework and added it to installed in settings.py but docker-compose up raises error . No module named rest_framework.Checked, everything is correct.Docker/virtualenv conflict ? May it be?

Docker and containerization might inspire the illusion that you do not need a virtual environment. distutil's Glpyh makes a very compelling argument against this misconception in this pycon talk.
The same fundamental aspects of virtualenv advantages apply for a container as they do for a non-containerized application, because fundamentally you're still running a linux distribution.
Debian and Red Hat are fantastically complex engineering projects.
Integrating billions of lines of C code.For example, you can just apt install libavcodec. Or yum install ffmpeg.
Writing a working build
system for one of those things is a PhD thesis. They integrate
thousands of Python packages simultaneously into one working
environment. They don't always tell you whether their tools use Python
or not.
And so, you might want to docker exec some tools inside a
container, they might be written in Python, if you sudo pip install
your application in there, now it's all broken.
So even in containers, isolate your application code from the system's
Regardless of whether you're using docker or not you should always run you application in a virtual environment.
Now in docker in particular using a virtualenv is a little trickier than it should be. Inside docker each RUN command runs in isolation and no state other than file system changes are kept from line to line. To install to a virutalenv you have to prepend the activation command on every line:
RUN apt-get install -y python-virtualenv
RUN virtualenv /appenv
RUN . /appenv/bin/activate; \
pip install -r requirements.txt
ENTRYPOINT . /appenv/bin/activate; \
run-the-app

A virtualenv is there for isolating the packages to a specific environment. Docker is also there to isolate the settings to a specific environment. So in essence if you use docker there isn't much benefit of using virtualenv too.
Just pip install thing into the docker environment directly it'll do no harm. To pip install the requirements use the dockerfile where you can execute commands.
You can find a pseudo code example below.
FROM /path/to/used/docker/image
RUN pip install -r requirements.txt