Develop Reference

Install Local Python Package with pip

I'm building a python package to use for 'global' functions (i.e. stuff that I will use in multiple other projects). I have built the package using py -m build and it then puts the MyPackage-0.1.0.tar.gz into the dist directory in my folder.
My goal is to be able to run pip install MyPackage from within any other projects, and it will install the latest build of my package. In other words, I do not want to use something like --find-links. This way, I could also include the package in a requirements.txt file.
I have tried putting the tarball in a directory which is on my system's PATH, and into a subfolder within there (e.g. PathDir/MyPackage/MyPackage-0.1.0.tar.gz), but I keep getting the same 'No matching distribution found' error.
The documentation for pip install says:
pip looks for packages in a number of places: on PyPI (if not disabled via --no-index), in the local filesystem, and in any additional repositories specified via --find-links or --index-url.
When it says 'in the local filesystem' where does it begin it's search? Is there a way to change this (e.g. set some environment variable)

When looking for files in the local filesystem, pip has no notion of search path. You must give a path accessible from the current working directory. It can be an absolute path:
pip install /path/to/MyPackage-0.1.0.tar.gz
a relative path:
cd /path
pip install to/MyPackage-0.1.0.tar.gz
or a simple name if the package file is inside the current working directory:
cd /path/to
pip install MyPackage-0.1.0.tar.gz

I found the answer after a lot of searching, and so here is the solution:
pip uses configuration files to define its internal settings. In these configuration files, you can specify default values for find-links. This means that python will look here for compatible packages, as well as online.
You can check what configurations have been set, and what files they will be searched in by running pip config list -v. You just need to edit/create one of the files listed and add your configuration as pip.ini with the following:
[install]
find-links=file://C:/Users/.../PathDir/MyPackage/
By creating this at the User/Global level (rather than the site level), this installation also works when inside a virtual environment.
Source: https://pip.pypa.io/en/stable/topics/configuration/

pip can install packages from the local file system. They do not need to be even package files, they can be just working directories or git checkouts.
Usually I use pip --editable:
pip --editable /path/to/my/python/package
With --editable changes in .py files in the folder are automatically reflected to your application.
You can use --editable in requirements.txt file as well.

"setup.py develop" installs git remote version instead of local version with live changes

Normally when I develop a Python package for personal use, I use python3 setup.py develop, and then perform pip3 install -e <path_to_package> within another virtualenv, allowing me to hack around with both at the same time. When I do gpip3 freeze I see the path to the package on my local machine:
-e /Users/myName/Documents/testpackage
When I store that package on GitHub and clone it back onto a local machine, I expect to be able to use setup.py develop the same way and keep developing the package on my local machine, regardless of whether or when I push back to GitHub. However, when I do gpip3 freeze, I see:
-e git+git#github.com:github_username/repo_name#-----latest_commit's_sha_code-----#egg=repo_name&subdirectory=xx/xx/testpackage
I would like my system to keep track of the local version instead of git's remote.
Note: I know how to commit and push local changes to GitHub and install the egg in local environments. My goal is to quickly test ideas with a development version of the package without continuously integrating.
Note 2: The GitHub address given in gpip3 freeze fails when I try it in an environment (FileNotFoundError: [Errno 2] No such file or directory: '/Users/myName/Documents/testenvironment/src/testpackage/setup.py')
But if I wanted pip3 to install the latest GitHub commit, I wouldn't be bothering with setup.py develop anyway.
Is there a way to signal to setup.py that I want it to ignore the remote in the cloned repo and pay attention only to the local path? Or is always referencing a remote when present the expected behavior of setup tools?
update :
The wording of the output in gpip3 freeze after python3 setup.py develop when a remote isn't present (below) leads me to consider that tracking a remote whenever possible may be the intended behavior :
# Editable Git install with no remote (testpackage ==0.0.1)
-e /Users/myName/Documents/testpackage
I have been working around this by git remote remove origin when I want my local changes to be reflected in local environments without pushing a new commit, though unideal for me.

My question was rooted in a misunderstanding of how to implement python3 setup.py develop.
My original method was :
1) python3 setup.py develop from within the package directory itself, which would install/link the egg globally
2) gpip3 freeze to get (I thought) the link to the egg (seeing all the extra git remote info here was confusing to me)
3) cd to another virtual environment, source bin/activate, then call pip3 install -e <link_copied_from_global_pip_freeze>
In fact there is no need to call python3 setup.py develop from within the package under development, or to use gpip3 freeze to get the egg link.
I can go directly to the virtual env and activate it, then use pip3 install -e <system_path_to_package_directory_containing_setup.py>. This will create an egg link in the package directory if it doesn't already exist. Edits within the package are reflected in the virtual environment as expected, and I can use Git version control freely within the package according to my needs without interference.
I assume there may be times to call python3 setup.py develop directly (setup.py develop --user also exists) but by not doing so I happen to avoid littering my global environment with extra packages.
Related info from 2014 question in the Python Disutils thread :
Questioner writes:
For years, I've been recommending:
$ python setup.py develop
[...]
Having said that, I also notice that:
$ pip install -e .
does the same thing.
Should I be recommending one over the other?
Noah answers :
You should recommend using pip for it, mostly because as you said that will work even with packages that don't use setuptools :-) It also is required when doing a develop install with extras, though that requires a slightly more verbose syntax due to a bug in pip.

What is the use case for `pip install -e`?

When I need to work on one of my pet projects, I simply clone the repository as usual (git clone <url>), edit what I need, run the tests, update the setup.py version, commit, push, build the packages and upload them to PyPI.
What is the advantage of using pip install -e? Should I be using it? How would it improve my workflow?

I find pip install -e extremely useful when simultaneously developing a product and a dependency, which I do a lot.
Example:
You build websites using Django for numerous clients, and have also developed an in-house Django app called locations which you reuse across many projects, so you make it available on pip and version it.
When you work on a project, you install the requirements as usual, which installs locations into site packages.
But you soon discover that locations could do with some improvements.
So you grab a copy of the locations repository and start making changes. Of course, you need to test these changes in the context of a Django project.
Simply go into your project and type:
pip install -e /path/to/locations/repo
This will overwrite the directory in site-packages with a symbolic link to the locations repository, meaning any changes to code in there will automatically be reflected - just reload the page (so long as you're using the development server).
The symbolic link looks at the current files in the directory, meaning you can switch branches to see changes or try different things etc...
The alternative would be to create a new version, push it to pip, and hope you've not forgotten anything. If you have many such in-house apps, this quickly becomes untenable.

For those who don't have time:
If you install your project with an -e flag (e.g. pip install -e mynumpy) and use it in your code (e.g. from mynumpy import some_function), when you make any change to some_function, you should be able to use the updated function without reinstalling it.

pip install -e is how setuptools dependencies are handled via pip.
What you typically do is to install the dependencies:
git clone URL
cd project
run pip install -e . or pip install -e .[dev]*
And now all the dependencies should be installed.
*[dev] is the name of the requirements group from setup.py
Other than setuptools (egg) there is also a wheel system of python installation.
Both these systems are based on promise that no building and compilation is performed.

What's the standard way to package a python project with dependencies?

I have a python project that has a few dependencies (defined under install_requires in setup.py). My ops people requires a package to be self contained and only depend on a python installation. The litmus test would be that they're able to get a zip-file and then unzip and run it without an internet connection.
Is there an easy way to package an install including dependencies? It is acceptable if I have to build on the OS/architecture that it will eventually be run on.
For what it's worth, I've tried both setup.py build and setup.py sdist, but they don't seem to fit the bill since they do not include dependencies. I've also considered virtualenv (which could be installed if absolutely necessary), but that has hard coded paths which makes it less than ideal.

There are a few nuances to how pip works. Unfortunately, using --prefix vendor to store all the dependencies of the project doesn't work if any of those dependencies, or dependencies of dependencies are installed into a place where pip can find them. It will skip those dependencies and just install the rest to your vendor folder.
In the past I've used virtualenv's --no-site-packages option to solve this issue. At one company we would ship the whole virtualenv, which includes the python binary. In the interest of only shipping the dependencies, you can combine using a virtualenv with the --prefix switch on pip to give yourself a clean environment that installs to the right place.
I'll provide an example script that creates a temporary virtualenv, activates it, then installs the dependencies to a local vendor folder. This is handy if you are running in CI.
#!/bin/bash
tempdir=$(mktemp -d -t project.XXX) # create a temporary directory
trap "rm -rf $tempdir" EXIT # ensure it is cleaned up
# create the virtualenv and exclude packages outside of it
virtualenv --python=$(which python2.7) --no-site-packages $tempdir/venv
# activate the virtualenv
source $tempdir/venv/bin/activate
# install the dependencies as above
pip install -r requirements.txt --prefix=vendor

In most cases you should be able to "vendor" all the dependencies. It's basically a crude version of virtualenv.
For example look at how the requests package includes chardet and urllib3 in its own source tree. Here's an example script that should do the initial downloading and copying for you: https://gist.github.com/proppy/1136723
Once you have the dependencies installed, you can reference them with from .some.namespace import dependency_name to make sure that you're using your local versions.

It's possible to do this with recent versions of pip (I'm using 8.1.2). On the build machine:
pip install -r requirements.txt --prefix vendor
Then run it:
PYTHONPATH=vendor/lib/python2.7/site-packages python yourapp.py
(This is basically an expansion of #valentjedi comment. Thanks!)

let's say you have python module app.py with dependencies in requirements.txt file.
first, install all your dependencies in appdeps folder.
python -m pip install -r requirements.txt --target=./appdeps
then in your app.py module add this dependency folder to the pythonpath
# app.py
import sys
sys.path.append('appdeps')
# rest of your module normally
#...
this will work the same way as if you were running this script from venv with all the dependencies installed inside ;>

Pip creates build/ directories

I use virtualenv to create isolated environments for my Python projects. Then i install dependencies with pip - Python package manager. Sometimes i forget to do source venv/bin/activate, and then pip creates build/ directories inside my projects. Why does pip create them? May i delete them, and if not, may i put them in my .hgignore file?
As far as i understand, pip stores source of downloaded packages there along a file called pip-delete-this-directory.txt. But when i delete it, everything still works, as the real code is being put into venv/lib/python2.7/site-packages/. Then what is build/ really for?

The build directory is where a packages gets unpacked into and build from. When the package is installed successfully, pip removes the unpacked dir from build, unless you've removed pip-delete-this-directory.txt. As described in pip-delete-this-directory.txt:
This file is placed here by pip to indicate the source was put
here by pip.
Once this package is successfully installed this source code will be
deleted (unless you remove this file).
Thus its less important for runtime environment. You could ignore it safely.
Also, you could use pip install -b customized_build_directory to specify another directory as build base directory, for example /tmp
Furthermore, you could pip install --no-download package_name to rebuild the package w/o downloading it, if the previous installation of the package failed.