Develop Reference

How to setup pip to download from mirror repository by default?

I am forced to download python packages from local mirror PyPi repository. I do this by using the -i and --trusted-host options. Whole installation command looks like this:
pip install -i https://sampleurl.com/pypi-remote/simple --trusted-host sample.host.com package_name
Having to type in that options each time is kinda annoying though (in reality those are long URL's). I've tried to create get_package.bat file (I'm working on Windows 10) with following content:
pip install -i https://sampleurl.com/pypi-remote/simple --trusted-host sample.host.com "%1"
It works perfectly fine, although when I wanted to execute pip search command, it turned out to be useless since it has hard-coded install command and there is no way to use it with search.
Is there any way in which I can setup pip to download from mirror repository by default, so that I can execute pip install package_name or pip search package_name without any additional options?
Eventually I could try making .bat file that would take 2 parameters like this:
pip %1 -i https://sampleurl.com/pypi-remote/simple --trusted-host sample.host.com "%2"
But I wonder if there's more "elegant" way to do this.

using pip config, on user or global level. I have /etc/pip.conf configured like this:
[global]
index=https://my-company/nexus/repository/pypi-group/pypi
index-url=https://my-company/nexus/repository/pypi-group/simple
trusted-host=my-company
but you can configure this using pip config on user or global level, something like:
pip config --user set global.index https://my-company/nexus/repository/pypi-group/pypi
pip config --user set global.index-url https://my-company/nexus/repository/pypi-group/simple
pip config --user set global.trusted-host my-company
#NOTES
--index-url is used by pip install
--index is used by pip search

Use pip3 config list -v to get list of locations where your pip.conf is located. Then go to one of the location(I prefer user) and add your URL.
The file should look like this, if empty then add the lines.
[global]
index-url=https://pypi.org/simple
extra-index-url=<your_url>
In case if you want pip to look into your URL first then switch the places of url on above options.
[global]
index-url=<your_url>
extra-index-url=https://pypi.org/simple

You could also add the -i and --trusted-host options to your requirements.txt like this:
-i https://sampleurl.com/pypi-remote/simple --trusted-host sample.host.com
numpy==1.23.4
pandas==1.5.1
And then, just do:
pip install -r requirements.txt
which gives:
Looking in indexes: https://sampleurl.com/pypi-remote/simple
Collecting numpy==1.23.4
Downloading numpy-1.23.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
...
I found this solution to work well in case you don't necessary want to create a pip.conf file (for example, in CIs).

Install python requirements.txt with Makefile only requirements.txt is changed

How can I run target make install only if requirements.txt is changed ?
I don't want to upgrade packages each time when I do make install
I found some workaround by creating fake file _requirements.txt.pyc but is ugly and dirty. It will refuse install pip requirements second time because requirements.txt has no changes
$ make install-pip-requirements
make: Nothing to be done for 'install-pip-requirements'.
But my goal is to do:
# first time,
$ make install # create virtual environment, install requirements
# second time
$ make install # detected and skipping creating virtual env,
# detect that requirements.txt have no changes
# and skipping installing again all python packages
make: Nothing to be done for 'install'.
Python package looks like:
.
├── Makefile
├── README.rst
├── lambda_handler.py
└── requirements.txt
I am using file, Makefile, for some automation in python:
/opt/virtual_env:
# create virtual env if folder not exists
python -m venv /opt/virtual_env
virtual: /opt/virtual_env
# if requirements.txt is modified than execute pip install
_requirements.txt.pyc: requirements.txt
/opt/virtual_env/bin/pip install -r --upgrade requirements.txt
echo > _requirements.txt.pyc
requirements: SOME MAGIG OR SOME make flags
pip install -r requirements.txt
install-pip-requirements: _requirements.txt.pyc
install: virtual requirements
I am sure that
Must be a better way
to do this;)

Not sure it will answer your question at this point. The better way is to use a fully fledged Python PIP project template.
We use cookiecutter to create a particular pip package with this cookiecutter template.
It has a Makefile, which does not constantly re-install all the dependencies and it makes use of Python tox, which allows running a project tests in different python envs automatically. You still can develop in dev virtualenv, but we update it only when new package is added, everything else is handle by tox.
But, what you show so far is trying to write a Python build from scratch, which was done with numerous project templates. If you really want to understand what is going on there, you can analyze these templates.
As followup: Because you expect it to work with a makefile, I'd suggest removing the --upgrade flag from the pip command. I suspect your requirements do not include versions that are needed for the project to work. We made an experience, that not putting versions there might badly brake things. Thus our requirements.txt looks like:
configure==0.5
falcon==0.3.0
futures==3.0.5
gevent==1.1.1
greenlet==0.4.9
gunicorn==19.4.5
hiredis==0.2.0
python-mimeparse==1.5.2
PyYAML==3.11
redis==2.10.5
six==1.10.0
eventlet==0.18.4
Using the requirements without --upgrade causes pip simply verify what is in virtualenv and what not. Everything that satisfies the required version will be skipped (no download). You can also reference git versions in requirements like that:
-e git+http://some-url-here/path-to/repository.git#branch-name-OR-commit-id#egg=package-name-how-to-appear-in-pip-freeze

#Andrei.Danciuc, make just needs two files to compare; you can use any of the output files from running pip install.
For example, I usually use a "vendored" folder, so I can alias the path to the "vendored" folder instead of using a dummy file.
# Only run install if requirements.txt is newer than vendored folder
vendored-folder := vendored
.PHONY: install
install: $(vendored-folder)
$(vendored-folder): requirements.txt
rm -rf $(vendored-folder)
pip install -r requirements.txt -t $(vendored-folder)
If you don't use a vendored folder, this code below should work for both virtualenv and global setups.
# Only run install if requirements.txt is newer than SITE_PACKAGES location
.PHONY: install
SITE_PACKAGES := $(shell pip show pip | grep '^Location' | cut -f2 -d':')
install: $(SITE_PACKAGES)
$(SITE_PACKAGES): requirements.txt
pip install -r requirements.txt

Specify extras_require with pip install -e

How can one manage to install extras_requires with pip when installing from a git repository ?
I know that you can do pip install project[extra] when the project is on pypi.
And you have to do pip install -e git+https://github.com/user/project.git#egg=project for a git repo but I didn't manage to find how to link these two options together.

This should work, per example #6
For remote repos:
pip install -e git+https://github.com/user/project.git#egg=project[extra]
And this for local ones (thanks to #Kurt-Bourbaki):
pip install -e .[extra]
As per #Jurt-Bourbaki:
If you are using zsh you need to escape square brackets or use quotes:
pip install -e .\[extra\]
# or
pip install -e ".[extra]"

Important to notice: you should not have whitespaces around or within brackets. I.e. this will work: -e ".[extra1,extra2]" but this won't: -e ". [extra1, extra2]" - and even as a row in requirements.txt file, where it is not so obvious. The worst thing about it is that when you have whitespace, extras are just silently ignored.

It may not be obvious for some users, and wasn't for me, so thought to highlight that extra in the following command
pip install -e ".[extra]"
needs to be replaced by the actual name of the extra requirements.
Example:
You add options.extras_require section to your setup.cfg as follows:
[options.extras_require]
test =
pre-commit>=2.10.1,<3.0
pylint>=2.7.2,<3.0
pytest>=6.2.2,<7.0
pytest-pspec>=0.0.4,<1.0
Then you install the test extra as follows
pip install -e ".[test]"

This also works when installing from a whl file so, for example, you can do:
pip install path/to/myapp-0.0.1-py3-none-any.whl[extra1]
This is very far from clear from the docs, and not particularly intuitive.

Using git + ssh to install packages with extras from private repositories:
pip install -e 'git+ssh://git#github.com/user/project.git#egg=project[extra1,extra2]'

Split requirements files in pip

To create Python virtual environments I use virtualenv and pip. The workflow is very simple:
$ virtualenv project
$ cd project
$ . bin/activate
$ pip install -r /path/to/requirements/req1.txt
$ pip install -r /path/to/requirements/req2.txt
The number of different requirement files can grow enough to make handy to have a way to include them at once, so I'd rather prefer to say:
$ pip install -r /path/to/requirements/req1_req2.txt
with req1_req2.txt containing something like:
include /path/to/requirements/req1.txt
include /path/to/requirements/req2.txt
or otherwise:
$ pip install -r /path/to/requirements/*.txt
None of that works and however much simple it could be, I can't figure out how to do what I want.
Any suggestion?

The -r flag isn't restricted to command-line use only, it can also be used inside requirements files. So running pip install -r req-1-and-2.txt when req-1-and-2.txt contains this:
-r req-1.txt
-r req-2.txt
will install everything specified in req-1.txt and req-2.txt.

Just on a note, you can also split the requirements based on your groupings and embed them in a single file ( or again can prepare multiple requirements file based on your environment), that you can execute.
For example, the test requirements here:
requirements-test.txt
pylint==2.4.4
pytest==5.3.2
The dev requirements here:
requirements-dev.txt
boto3>=1.12.11
Master requirements file containing your other requirements:
requirements.txt
-r requirements-dev.txt
-r requirements-test.txt
Now, you can just install the requirements file embedding your other requirements
pip3 install -r requirements.txt

How to point pip at a Mercurial branch?

I'm trying to install my application via pip to a virtualenv for testing.
Works fine for installing the default or tip like so:
pip install -e hg+https://username#bitbucket.org/username/app_name#egg=app_name
But is there any way to point to a branch, rather than just getting the tip. Not sure if this would be a mercurial thing, bitbucket, or pip.
Bitbucket allows for downloading of a tagged version of the code, but I can only get it to work while logged into the browser. I tried installing from a tag tar.gz like so:
pip install https://username#bitbucket.org/username/app_name/get/bbc4286a75db.tar.gz
but even after entering my password it returns a 401 Unauthorized (Its a Private Repo)

In official pip documentation in section VCS Support:
Mercurial
The supported schemes are: hg+http, hg+https, hg+static-http and
hg+ssh:
-e hg+http://hg.myproject.org/MyProject/#egg=MyProject
-e hg+https://hg.myproject.org/MyProject/#egg=MyProject
-e hg+ssh://hg#myproject.org/MyProject/#egg=MyProject
You can also specify a revision number, a revision hash, a tag name or
a local branch name:
-e hg+http://hg.myproject.org/MyProject/#da39a3ee5e6b#egg=MyProject
-e hg+http://hg.myproject.org/MyProject/#2019#egg=MyProject
-e hg+http://hg.myproject.org/MyProject/#v1.0#egg=MyProject
-e hg+http://hg.myproject.org/MyProject/#special_feature#egg=MyProject
The syntax is the same when specifying repo at the command line
pip install -e hg+http://hg.myproject.org/MyProject/#special_feature#egg=MyProject
and it works when not using -e option starting from version 0.8.2.