If i type pip freeze > requirements.txt, the resulting file looks similar to this:
argparse==1.2.1
h5py==2.2.0
wsgiref==0.1.2
Some libraries are under ongoing development. This happened to me regarding h5py, which is now (as of this writing) available in version 2.2.1. Thus, using pip install -r requirements.txt throws an error, saying version 2.2.0 of h5py was not found:
No distributions matching the version for h5py==2.2.0 (from -r requirements.txt (line 2))
Is it considered good practice to maintain the requirements via pip freeze at all? Obviously, I can not rely on specific version numbers being still available in the future. I would like to deploy my applications in the future, even if they are several years old, without compatibility problems regarding version numbers. Is there a way to make the output of pip freeze future-safe?
I thought about manipulating the output file of pip freeze by using the greater than symbol >= instead of the equals symbol ==, so the output would look like the following:
argparse>=1.2.1
h5py>=2.2.0
wsgiref>=0.1.2
But I can imagine that this will break my applications if any of the libraries breaks backward-compatibility in a future version.
To answer the first question, yes, it is pretty common to use pip freeze to manage requirements. If your project is packaged you can also set dependencies directly in the setup.py file.
You can set the requirements to greater than or equal to version x, but as you speculate, this can turn around and bite you if a dependency makes changes to their api that break your required functionality. You can also ensure that an installed dependency is less than a certain version. i.e. If you're on version 1.0 of a package and would like minor updates but a major release scares you (whether its released yet or not) you can require example>=1.0.0,<2.0.0
More info on requirements files
In the end, pip freeze is just a tool to show you what you currently have installed, it doesn't know, or care, if it works for you. What you use to replicate environments based on this data also doesn't really matter; version conflicts, updates breaking backwards compatibility, bugs and other such issues in dependencies will (at least once) cause you grief. Keeping tabs on the state of your project's major dependencies and doing automated testing with new versions will save you a lot of time and headache (or at least headache).
Related
I have come across some nasty dependencies.
Looking around I found solutions like upgrade this, downgrade that...
Some solutions work for some but not for others.
Is there a more 'rounded' way to tackle this issue?
To test this:
I wrote a script to create a virtual environment, attempt to install a requirements file, then delete this environment which I run to quickly test if changes to requirements result in a successful installation or not.
I wrote a "test app" which uses the basic functionalities of all the libraries I use in one place, to see if despite a successful installation, there are dependencies that pip is unaware of (due to problematic 3rd party library architecture) that break. I can edit the dependencies of this app, then commit to run build actions that run it and see if it completes or crashes.
This way I can upgrade and add libraries more rapidly.
If all libraries used semantic versioning, declared all their dependencies via requirement files and did not define upper versions in requirement files, this would be a lot simpler. Unfortunately, one of the DB-vendors I use (which I shall not name) is very problematic and has a problematic Python library (amongst many other problems).
First you need to understand that pip can resolve problems one at a time and when you put it in a corner, it can't go further.
But, if you give to pip the 'big problem' it has a nice way to try to resolve it. It may not always work, but for most cases it will.
The solutions you normally find out there are in some cases a coincidence. Someone has an environment similar to another person and one solution works for both.
But if the solution takes into consideration 'your present environment' then the solution should work for more people than just 'the coincidences'.
Disclaimer: below are linux/terminal commands.
Upgrade pip. We want the smartest pip we can get.
pip install --upgrade pip
Extract the list of packages you want to install.
In my case (these and many others, trimmed for brevity)
google-cloud-texttospeech attrdict google-cloud-language transformers
Give them all at once to pip.
pip install google-cloud-texttospeech attrdict google-cloud-language transformers
It will try all the combinations of versions and dependencies' versions until it finds something suitable. This will potentially download a ton of packages just to see their dependencies, so you only want to make this once.
If happy with the result, extract the requirements file.
pip freeze > requirements.txt
This contains all the packages installed, we are not interested in all.
And from it, extract the specific versions of your desired packages.
cat requirements.txt | egrep -i "google-cloud-texttospeech|attrdict|google-cloud-language|transformers"
Note: The command above may not list all your required packages, just the ones that appear in the requirements.txt, so check that all show up.
attrdict==2.0.1
google-cloud-language==1.2.0
google-cloud-texttospeech==2.12.3
transformers==2.11.0
Now you can put that on a file like resolved-dependencies.txt
And next time, install the packages directly with the valid & compatible version with.
pip install -r resolved-dependencies.txt
I have developers that want to be able to publish a library as a "beta" or "release" version.
i.e.:
1.2.3-beta
1.2.3
On the consuming project, they couldn't give me any specific criteria of when they would want to use a beta or release package. We have CI in place, but without any definitive "when" I can't support two separate pip feeds, as they may flip flop. So I've suggested taking advantage of version range syntax in the requirements file, so they can just specify during checkin what they want. They've never had to do anything like this, and I'm basically a python rookie. Is it possible to filter on pre-release labels? I.e.
Will lib == 1.*.*-* pick up a beta package?
and
Will lib == 1.*.*, !=1.*.*-* pick up a release package and be sure to exclude any beta packages?
I would try out my theory myself, but I don't know python well enough to mock up some sort of sample libs locally, and they're too busy to research it.
By default pip will not install prereleases, like 1.0.0.b1
To enable installation of prereleases, you use the --pre flag with pip.
You can also use prerelease version specifiers to force pip to consider prereleases for individual packages without needing to use --pre. From https://pip.pypa.io/en/stable/reference/pip_install/#pre-release-versions:
If a Requirement specifier includes a pre-release or development version (e.g. >=0.0.dev0) then pip will allow pre-release and development versions for that requirement.
So in your requirements.txt file, you'd have something like:
package_a>=1.2.3 # will not install pre-releases
package_b>=1.2.3.dev0 # will install pre-releaes
I have a requirements.txt file for a Python code base. The file has everything specified:
pytz==2017.2
requests==2.18.4
six==1.11.0
I am adding a new package. Should I list its version? If yes, how do I pick a version to specify?
Check out the pip docs for more info, but basically you do not need to specify a version. Doing so can avoid headaches though, as specifying a version allows you to guarantee you do not end up in dependency hell.
Note that if you are creating a package to be deployed and pip-installed, you should use the install-requires metadata instead of relying on requirements.txt.
Also, it's a good idea to get into the habit of using virtual environments to avoid dependency issues, especially when developing your own stuff. Anaconda offers a simple solution with the conda create command, and virtualenv works great with virtualenvwrapper for a lighter-weight solution. Another solution, pipenv, is quite popular.
Specifying a version is not a requirement though it does help a lot in the future. Some versions of packages will not work well with other packages and their respective versions. It is hard to predict how changes in the future will effect these interrelationships. This is why it is very beneficial to create a snapshot in time (in your requirements.txt) showing which version interrelationships do work.
To create a requirements.txt file including the versions of the packages that you're using do the following. In your console/ terminal cd into the location that you would like your requirement.txt to be and enter:
pip freeze > requirements.txt
This will automatically generate a requirement.txt file including the packages that you have installed with their respective versions.
A tip - you should aim to be using a virtual environment for each individual project that you'll be working on. This creates a 'bubble' for you to work within and to install specific package versions in, without it effecting your other projects. It will save you a lot of headaches in the future as your packages and versions will be kept project specific. I suggest using Anacondas virtual environment.
No, there is no need to specify a version. It's probably a good idea to specify one, though.
If you want to specify a version but you don't know which version to specify, try using pip freeze, which will dump out a list of all the packages you currently have installed and what their versions are.
1 I get the warning msg when I pip install my Django project requirements
wechat-sdk 0.6.4 has requirement requests==2.6.0, but you'll have
requests 2.9.1 which is incompatible.
2 then follow the tips I uninstall requests and install proper version, but I get another warning
python-social-auth 0.2.21 has requirement requests>=2.9.1, but you'll
have requests 2.6.0 which is incompatible.
So, I'm trapped in the endless loop
Can anyone give any advice?
As far as I can see you have the following options:
Run pip with the --no-dependencies argument and hope it will just work. pip install wechatpy --no-dependencies (or whatever the package is called, I am not familiar with it) for example. This ignores the dependencies when installing. Maybe the requirements are outdated and this will let you proceed. For this you most likely want to satisfy the requests>=2.9.1 requirement so you should install python-social-auth normally and then try the other package without dependencies.
Look for older versions of the packages you are installing that have compatible requirements. Depending on the setup of your project this might not be possible to do because you need features of the later versions (or the old versions might be insecure).
You can try patching one of your requirements locally (download from the source, change the code to make it work with the conflicting requests version). And then import the local version of the packages. Remember to remove the requirement from your project's requirements.txt in this case to stop other people working on / using your project from running into the same issue, and include the local version as part of the project (track it on Git). Check the license of the packages you are modifying to see if you are allowed to modify and redistribute them. (Optional: Make a pull request on the packages' Github with your change(s) so other people can benefit from them)
Replace one or both of the packages by something else. They might just not be compatible or using a local, modified version might not be viable.
I am using requirement.txt to specify the package dependencies that are used in my python application. And everything seems to work fine for packages of which either there are no internal dependencies or for the one using the package dependencies which are not already installed.
The issue occurs when i try to install a package which has a nested dependency on some other package and an older version of this package is already installed.
I know i can avoid this while installing a package manually bu using pip install -U --no-deps <package_name>. I want to understand how to do this using the requirement.txt as the deployment and requirement installation is an automated process.
Note:
The already installed package is not something i am directly using in my project but is part of a different project on the same server.
Thanks in advance.
Dependency resolution is a fairly complicated problem. A requirements.txt just specifies your dependencies with optional version ranges. If you want to "lock" your transitive dependencies (dependencies of dependencies) in place you would have to produce a requirements.txt that contains exact versions of every package you install with something like pip freeze. This doesn't solve the problem but it would at least point out to you on an install which dependencies conflict so that you can manually pick the right versions.
That being said the new (as of writing) officially supported tool for managing application dependencies is Pipenv. This tool will both manage the exact versions of transitive dependencies for you (so you won't have to maintain a "requirements.txt" manually) and it will isolate the packages that your code requires from the rest of the system. (It does this using the virtualenv tool under the hood). This isolation should fix your problems with breaking a colocated project since your project can have different versions of libraries than the rest of the system.
(TL;DR Try using Pipenv and see if your problem just disappears)