Develop Reference

pipenv deleted libraries from my virtual environment?

I am using Homebrew-installed pipenv to manage my virtual environments for Python projects. I navigate to my Python project's folder and use the pipenv shell command to activate the venv.
It has worked fine, until today when I noticed that I can't run my app.py from within the shell using the python3 app.py command. I get the ModuleNotFoundError: No module named 'flask' right from line 1.
When I run which python3 and which pip3, I see the expected response that specifies that I'm within my venv. When I run pip3 list, I only see pip, setuptools and wheel.
This is odd, because just very recently everything has worked fine (1-2 weeks ago?), and I'm positive that I personally didn't do anything that would mess with the libraries/requirements.
The Pipfile still lists all the requirements as expected. So how come they got deleted from my virtual environment?
I understand that I can just redownload all of the requirements; I'm just curious about why this happened in the first place.
UPDATE: I just realised that I did change the name of the folder which contains the project; I assume this is the cause. Since I've redownloaded the requirements already, does that mean I now have duplicates existing somewhere? If so, where?

If you moved/renamed the folder where you created your virtual env, then the next time you try to activate the virtual env there, Pipenv will create a brand new virtual env. This is because Pipenv creates the actual virtual env folders based on the full path to the project directory. This is noted in the docs:
https://pipenv-fork.readthedocs.io/en/latest/install.html#virtualenv-mapping-caveat
Pipenv automatically maps projects to their specific virtualenvs.
The virtualenv is stored globally with the name of the project’s root directory plus the hash of the full path to the project’s root (e.g., my_project-a3de50).
If you change your project’s path, you break such a default mapping and pipenv will no longer be able to find and to use the project’s virtualenv.
Emphasis on the 3rd bullet. So it didn't delete your packages, it basically created a new one. You should have also seen a notice that it was creating a new one:
demo$ pipenv shell
Launching subshell in virtual environment...
...
(demo) demo$
exit
demo$ cd ..
~$ mv demo demo2
~$ cd demo2
demo2$ pipenv shell
Creating a virtualenv for this project...
...
(demo2) demo2$
That "Creating a virtualenv..." means it's creating a new one.
Now, on to:
does that mean I now have duplicates existing somewhere? If so, where?
It means you still have your previous virtual env folder somewhere, where you previously installed your packages. You can try using the --venv option to get the top-level directory where Pipenv creates all virtual env folders. In your new env:
(demo2) demo2$ pipenv --venv
/Users/gino.mempin/.venvs/demo2-4Y1NLH_X
As mentioned, the virtual env folder here is demo2-4Y1NLH_X, and the top-level folder is (for my case) .venvs. The default is something like /.local/share/ or whatever you set WORKON_HOME to (see Custom Virtual Environment Location). Just run the --venv for yourself.
You can try going there, and it will list all the virtual envs you have created:
(demo2) demo2$ ls /Users/gino.mempin/.venvs
demo-tSf-ZA7f
demo2-4Y1NLH_X
some-other-project-ABJaje5
another-project-8WUmE08m
...
Here, if you are lucky, you can find the name of your old folder, and then simply delete it if you want to cleanup. If you are unlucky, there'll be multiple folders with the same name, and you won't be able to tell which one was your old folder.
(demo2) demo2$ ls /Users/gino.mempin/.venvs
demo-tSf-ZA7f
demo-7I2ki6rH
demo-8WUmE08m
demo2-4Y1NLH_X
There is currently no way to get the full path to the original directly from the virtual env folder-hash itself. (See related: How to remove all pipenv virtualenvs when the directory was deleted?). There is also no way to reuse your old virtual env and copy it to your new one. But you don't need to anyway, creating virtual envs is inexpensive, just recreate it and reinstall all previous packages.

pipenv : how to force virtualenv directory?

Actually, pipenv will install the virtualenv with a path like this :
$WORKON_HOME/<base_dir>-<hash>
Is it possible to have exactly the path I want, that is without the base_dir and the hash, for exemple :
/home/user/myapp_venv

Apart from using a custom location, you can also install the virtualenv in your project's directory. Just add the following line in your .bashrc/.zshrc file:
export PIPENV_VENV_IN_PROJECT=1
Just wanted to let others know that there is another approach available too.
Should you keep the virtualenv inside or outside the project's directory is an opinionated question afterall.

There is an undocumented feature of pipenv, it could locate virtualenv path from VIRTUAL_ENV environment variable, but you need to create virtualenv manually:
virtualenv /home/user/myapp_venv
VIRTUAL_ENV=/home/user/myapp_venv pipenv install

There's an undocumented feature in pipenv: if you create a file named .venv in the project root with a path in it, pipenv will use that instead of an autogenerated path.
This, however, is more fit for cases when you already have an established set of environments that you wish to reuse. Otherwise, placing environments in arbitrary places is prone to create a mess eventually. pipenv relieves you from this task specifically to keep them all in one predictable place and eliminate accidental collisions from human error.

In windows, Pycharm user by
make .venv fold in project
Settings->Project->Project interpreter->pipenv Environment
it works for me

Creating a directory .venv in the project then try to initiate with pipenv install <some package> did not work for me at the first in my mac, it was always creating virtual environment somewhere else. Though later I figure it out the reason. The hack is, first cd to the project folder cd /Home/.../MyProject then create a empty Pipfile touch Pipfile the create a directory mkdir .venv should work. Now you can run pipenv install and .venv folder will be used.
Now the elaboration is given form the source code. When the pipenv try to create create a virtual env it looks to the directory dot_venv = os.path.join(self.project_directory, ".venv") (taken from source code) and thats how the self.project_directory looks like:
#property
def project_directory(self):
# type: () -> str
return os.path.abspath(os.path.join(self.pipfile_location, os.pardir))
So that function looks for the pipfile location to create the path it will create the virtual environment. So if you create the empty Pipfile before hand it is confirmed that it will find that file and then find the .venv directory and create the path all together.

Why is my virtualenv's pip listing packages in my lib/python2.7 directory?

In my home, I have a directory named lib/python2.7 (there are actually five directories like that, for different python versions). Since this is a shared hosting (Webfaction), that directory is fundamental to me. There, I have stuff like virtualenv and virtualenvwrapper installed, since as customer of a shared hosting, I have no access to sudo and installing global packages.
However, when I create a virtualenv:
$ mkvirtualenv myenvironment
$ workon myenvironment
$ which pip
# outputs the myenvironment's path to pip
$ pip freeze
The command shows the whole packages list under my lib/python2.7 (this includes the same virtualenv packages, and conflicting packages I have due to... legacy... reasons). This also annoys me if I want to install a package which is the name of a package in lib/python2.7 since it does not allow me to update it.
Right inside the workon environment, I try to check whether the PYTHONPATH has weird stuff, but it is empty:
$ echo $PYTHONPATH
# shows a blank line
It is also empty if I try that command out of any virtual environment.
It seems that --no-site-packages is default but solves just part of the problem. This means: pip freeze | wc -l displays a lesser value when in an environment than when executing globally, out of any environment, which tells me that there are certain already-provided packages that are being excluded and are from the hosting itself (and not installed by me since, again, the hosting is shared and I don't have access to global space).
My question is: How can I solve this? I want my virtualenv not list the packages in $HOME/lib/python2.7
Please avoid dupe-linking to this question, nothing was useful there and still does not have an accepted answer. I wrote this question after reading and trying each solution in that question

I think you need to specify python version. You can specify python version with which you want to create virtual environment using command like
virtualenv -p /usr/bin/python3.4 virt/virtname --no-site-packages
Because when you not specify a python version, virtualenv creates a environment with pythonv2.7 and hence all packages end up in the folder you mentioned.

Found the solution after deeply digging. This is a Webfaction custom but this could apply to any installation like this if the problem occurs.
The core of the problem is that Webfaction configured a sitecustomize.py file for us. The file is located at /usr/local/lib/pythonX.Y/sitecustomize.py and adds by itself the contents of ~/lib/pythonX.Y (and conditionally the contents of any python app under ~/webapps if you are working under a directory of it to run a python script).
Even when the virtualenv's python executable is a different one, it will load the said sitecustomize.py file each time it runs as the base python executable does.
The workaround here? Create an empty sitecustomize.py in your virtualenv to override the other:
touch ~/.virtualenvs/yourvenv/lib/pythonX.Y/sitecustomize.py
And it will work. Take this as reference if you are stuck here like I was
Notes: Replace X.Y on each case with the corresponding version you are working. Additionally remember: You cannot remove or edit the base sitecustomize.py since you are in a shared hosting, in this case. However, overriding will work for each case as long as you do this for every virtualenv you want.

Virtual Environments with Python

Hi I've been reading a lot about virtual environments but I don't seem to get one thing.
So I have my path like this:
../my_app/
../my_app/app.py
..etc
Where should I put the virtual environment?
Inside my_appas /my_app/venv/bin,include,lib?
or at the same level as my_app
/my_app/
/venv/
I don't understand if the location matters or if by using activate it will reference it instead of going to the main environment.
I hope this question makes sense.
Thanks!

I recommend utilizing the root directory which virtualenv creates as the root directory for your source files. Virtual-envirments are designed to be tied to a project, not shared between different projects. Example, lets say I write a lot of code in my ~/school directory. I'd cd to ~/school, do virtualenv ENV. Now I have an ENV directory in which to keep my source files and dependencies for the project. So you can create a ~/school/ENV/source folder in which to keep all your source folders. And all your virtual-environment files are close to your program, easily accessible in the ENV directory.
EDIT:
To address one part of your question: As long as you keep track of your environment, and you source bin/activate before running your python programs and installing dependencies with pip, you can install your virtual environment anywhere.

I don't understand if the location matters or if by using activate it
will reference it instead of going to the main environment.
It doesn't matter, as activate will take care of the paths correctly, but you shouldn't put your virtual environment in your source, as it is not part of your application (and its not portable). Instead, create a file with your requirements and put that under your source control.
You should put it in any directory other than your source code. The activate script will make sure the paths point to the right places.
Here is an example:
$ virtualenv /home/somedir/envs/myenv
... # some output
$ source /home/somedir/envs/myenv/bin/activate
(myenv) $ mkdir /home/somedir/projects
(myenv) $ cd /home/somedir/projects
(myenv) projects $
As you can see, the virtual environment is in the envs directory and is called myenv. Your source is in /home/somedir/projects. Type deactivate to exit your virtual environment.

How do I create a project without the project folder?

I'm new to pyramid and paster, just reading the docs for now. I use virtualenv and inside the virtualenv dir I want to start a pyramid project. The problem is that I would like for paster to not create a dir with the project name, and instead put all the scaffold files on the current dir (the venv root).
I thought about just not using paster but I still wouldn't know how to point to my app on development.ini "use" option.
I could also have my virtualenv on an entirely different place of my filesystem, but that seems weird to me (maybe virtualenvwrapper could make it easier). Any other way to do this?

It is confusing at first but your code really doesn't need to be in your virtual environment directory at all. Actually it's better not to put your code inside your environment, as you might want to use different environments with the same code, for example to test your code with different versions of Python or different versions of a library.
virtualenvwrapper does put all your environments in a single place. virtualenvwrapper is a convenient tool on top of virtualenv but you don't need it to put your code and your environments in different places. Maybe you should get a bit more comfortable with virtualenv itself before starting to use virtualenvwrapper.
You should let paster create a directory with the project name. This is the directory that you will commit in version control (eg. git, mercurial...). You don't want to commit the directory containing the virtual environment.

This is really just bike shedding because how you create the project and the virtualenv are irrelevant and you can place either of them anywhere, including within each other.
However, if you really want to, you can paster create -t pyramid_starter -o .. <current_directory_name> to create the project within the current directory.
To create a new project:
cd ~/work/my_repo
virtualenv --no-site-packages env
env/bin/pip install pyramid
env/bin/paster create -t pyramid_starter -o .. my_repo
git init
echo 'env' > .gitignore
git add .
I'll usually do this when setting up a new machine:
cd ~/work
git clone /path/to/<my repo>.git
cd my_repo
virtualenv --no-site-packages env
env/bin/pip install -e . # equivalent to env/bin/python setup.py develop
Using the setup I just mentioned, you'd want to add the env directory to your .gitignore file.