Develop Reference

Is there an alternative to pip freeze in order to keep packages up to date?

So I am setting up a Github repository with Gitpod through my Ipad. I learned I could use this command in order to easily load every requirement with the .gitpod.yml file.
pip freeze > requirements.txt
My question is: inside the requirements file, the packages are listed with their current version, but I would love them all to be of the form:
pandas==*
and not:
pandas==1.4.1
Thanks in advance for any help!

You can do:
pip freeze | sed 's|==.*|==*|g' > requirements.txt

inside the requirements file, the packages are listed with their current version
...keep packages up to date?
Then put the entry as just pandas without any version number:
pandas
It will upgrade to the latest version whenever you do pip install --upgrade -r requirements.txt.
Note that you shouldn't do this in a production environment because it's possible for a future version to become incompatible with your current code or other libraries.

Make pip install modules even if one fails

I'm using pip to install modules from a requirements file produced with pip freeze. However the problem sometimes it's unable to install or download one module and then everything fails and doesn't install anything. Is there a way to make it install the modules that satisfy the requirements?

With pip only, I would say no. pip and Python packages generally are designed to work in such a way that you might need dependencies installed in order to install the package itself. Thus, they don't have an option to try despite of failures.
However, pip install -r requirements.txt simply goes through the file line-by-line. You can iterate the every single item yourself and call pip install for it, without caring the result (was the installation successfully or not). With shell scripting this could be done e.g.:
cat requirements.txt|xargs pip install
The example does not understand comments, spaces, etc. so you might need to how something more complex in place for a real-life scenario.
Alternative you can simply run pip in loop until it gives a successful return value.
But as a real solution I would recommend you to set up your own Python package mirror server, or a local cache - which would be another question.

Pip install from a specific commit prompts "requirements already satisfied"

I'm using pip and a requirements.txt file to handle my python packages in my virtualenv. I have a particular package I install from Github so that inside my file I have:
git+ssh://git#github.com/myuser/mypackage.git#egg=mypackage
Since I'm working on the package quite often I need to re-install it but:
pip install -r requirements.txt gives me back
Requirement already satisfied (use --upgrade to upgrade)...
for all the packages in requirements.txt that have new versions.
If I run pip install -r requirements.txt --upgrade it tries to upgrade all my packages (that I do NOT want) but I want to upgrade only mypackage. In requirements.txt I've tried to add a specific commit, like so:
git+ssh://git#github.com/myuser/mypackage.git#733c5b616da27cba14478c24b#egg=mypackage
But when I run pip again it throws:
Requirement already satisfied (use --upgrade to upgrade)..bla bla bla
QUESTION:
Is there a way to upgrade only the specific package mypackage possibily using the requirements.txt file?
Do I need to specify the #egg=mypackage?

The reason you're getting Requirement already satisfied is because if you do not pass --upgrade or -U (the shorthand), the package is not modified if it is already installed.
(This part of the command has had a lot of discussion. Check out the first 4 issues here)
Is there a way to upgrade only the specific package mypackage possibily using the requirements.txt file?
You need to specify just mypackage to pip when telling it to upgrade. If you wanted to update only requests, the pip command is:
pip install --upgrade requests
Similarly, to update from your git repository, you want to do:
pip install --upgrade git+ssh://git#github.com/myuser/mypackage.git#egg=mypackage
Since it's a URL is a long thing, what I suggest you do what #daphtdazz suggests, use multiple requirements files, as follows:
requirements.txt
requests~=2.12.3
simplejson~=3.10.0
-r git_requirements.txt
git_requirements.txt
git+ssh://git#github.com/myuser/mypackage.git#egg=mypackage
Additionally, I suggest you use shell-aliases for your shell to ease the typing load.
alias pip_git_upgrade="pip install --upgrade -r git_requirements.txt"
Do I need to specify the #egg=mypackage?
To quote from pip's official documentation:
Any URL may use the #egg=name syntax to explicitly state the project name.
Basically, using #egg=mypackage is a good idea since you are making the the project name explicit.

If you have dependencies that need to be at a particular version, then you should fix them in your requirements file to stay at that version. So for example (although not realistic):
mock~=2.0.0
pexpect==2.4.1
git+ssh://git#github.com/myuser/mypackage.git#733c5b616da27cba14478c24b#egg=mypackage
mock will be updated to any version that looks like 2.0.* (normally changes in the most minor number are bugfixes, so you generally want this)
pexpect will be fixed at 2.4.1
mypackage will always be updated whenever possible.
If you only want to upgrade a single package though, then just upgrade that one:
pip install -U git+ssh://git#github.com/myuser/mypackage.git
Another alternative if you want to upgrade all of them regularly but some more regularly than others would be to split up the requirements file. See the pip docs. I suspect this needs an up to date version of pip and setuptools (but you're updating those regularly anyway, right??).
For example, you could then have:
update_regularly_reqs.txt
git+ssh://git#github.com/myuser/mypackage.git#733c5b616da27cba14478c24b#egg=mypackage
all_requirements.txt
-r update_regularly_reqs.txt
mock~=2.0.0
pexpect==2.4.1
Edit to add info on #egg=
The #egg=mypackage bit is required if you want to check it out using pip and also edit the code in that package, but then you need to use:
-e git+ssh://...#egg=mypackage
pip will then make a directory in the src directory in your virtualenv's home directory (use cdvirtualenv to find it) with that name, or at least it did on my system, and will check out the code using git clone (or appropriate for Mercurial or SVN if using those) so you can go and edit it in place.
But if you don't specify -e (as you did) then I think it checks it out as a normal package, which makes it harder for you to manage if you want to edit it in place, and then you don't need the #egg= bit.
No doubt there are lots of config options too... a good place to start is that doc I linked.

What's the difference between direct pip install and the requirements.txt?

I'm confused. I've got a working pip install command (meaning: it installs a version of a library from Github which works for me), and I have a non-working (meaning: it installs a version of a library which does not work for me) way of putting that requirement into a requirements.txt file.
More concrete:
If I type on the command line
pip install -e 'git://github.com/mozilla/elasticutils.git#egg=elasticutils'
and then test my program, all works fine. If I put this line into my requirements.txt:
-e git://github.com/mozilla/elasticutils.git#egg=elasticutils
and then run my program, it breaks with an error (only the library should have changed, so I guess sth has changed in that library between the two versions).
But shouldn't both versions do exactly the same?? (Of course I've done my best to remove the installed version of the library between the two tests again, using pip uninstall elasticutils.)
Any information welcome …

Yep, as I wrote in my comment above, there seems to be a dependency-override when the requirements.txt states different than the dependencies in the packages. In my case installing the package manually also installed the (newer) version of requests, namely 1.2.0. Using the requirements.txt always installed (due to the override) the version 0.14.2 of requests.
Problem solved by updating the requests version in the requirements.txt :-)

Well I don't know exactly what's the difference, but when I want something to be installed from the requirements.txt and it's a git repo I do the following line:
#git+https://github.com/user/package_name.git
and then installing as following:
pip install -r requirements.txt

How can I get pip install's -I flag to work with a requirements file?

I feel like there must be a way to do this, but for the life of me I can't figure out how: I want to run pip against a requirements file in a virtualenv so that no matter what packages are in the virtualenv before I run pip, the requirements file is totally fulfilled (including specific versions) after I run it.
The problem now is that if I have an older version of a package installed in the virtualenv than is listed in the requirements file, it complains about the version mismatch and exits (it should just update the package to the given version). The command I'm running is pip install -I -r requirements.txt and according to pip's help, -I is supposed to make pip "Ignore the installed packages (reinstalling instead)" but it definitely isn't doing that.
What am I missing?
(It'd be nice if pip skipped the packages that are already fulfilled too.)

I figured out what the cause of my pip problems was. Long story short, source left over in the virtualenv's build directory was causing an error that made packages upgrades fail. What I actually should have been doing was clearing out that directory (which pip doesn't always do I guess) before running the pip install and it seems to do everything I want after when paired with the --upgrade/-U flag.