I need to specify a version for a package that one of my dependencies depends on, but my package does not directly depend on. Say my CI is trying to build package_a, it depends on package_b and does not depend on package_c. But package_b depends on package_c and it needs to be a particular version. In other words, "don't install this, but if you end up installing this, please install this version."
This appears to be doable in conda with run_constrainted but I cannot determine if it is possible to do in pip in the requirements.
A less appealing workaround would just be to add this other package (package_c) as another requirement.
Related
I have my own pip package, which is called my_package.
It's quite large, and I want to split it into a few packages, which I can optionally install if necessary.
So my goal is: to have main package called my_package, and some extra packages (for example, my_package[extra1]), which can extend functionality of the main package.
I've done some research, and found out that you can specify [project.optional-dependencies] in pyproject.toml file, and it will install neccessary optional dependencies to your project.
And it seems like a good idea: to make an optional functionality as a separate pip package.
But the question is: how to gracefully implement it into the main package?
I mean, I want to install this "extra" package by pip install my_package[extra1], and it should lead to automatically added functionality to my project.
Under the hood it just installes a different package, but how to use it by the main package?
Should I check the dependency existance in environment by some way?
I have come across some nasty dependencies.
Looking around I found solutions like upgrade this, downgrade that...
Some solutions work for some but not for others.
Is there a more 'rounded' way to tackle this issue?
To test this:
I wrote a script to create a virtual environment, attempt to install a requirements file, then delete this environment which I run to quickly test if changes to requirements result in a successful installation or not.
I wrote a "test app" which uses the basic functionalities of all the libraries I use in one place, to see if despite a successful installation, there are dependencies that pip is unaware of (due to problematic 3rd party library architecture) that break. I can edit the dependencies of this app, then commit to run build actions that run it and see if it completes or crashes.
This way I can upgrade and add libraries more rapidly.
If all libraries used semantic versioning, declared all their dependencies via requirement files and did not define upper versions in requirement files, this would be a lot simpler. Unfortunately, one of the DB-vendors I use (which I shall not name) is very problematic and has a problematic Python library (amongst many other problems).
First you need to understand that pip can resolve problems one at a time and when you put it in a corner, it can't go further.
But, if you give to pip the 'big problem' it has a nice way to try to resolve it. It may not always work, but for most cases it will.
The solutions you normally find out there are in some cases a coincidence. Someone has an environment similar to another person and one solution works for both.
But if the solution takes into consideration 'your present environment' then the solution should work for more people than just 'the coincidences'.
Disclaimer: below are linux/terminal commands.
Upgrade pip. We want the smartest pip we can get.
pip install --upgrade pip
Extract the list of packages you want to install.
In my case (these and many others, trimmed for brevity)
google-cloud-texttospeech attrdict google-cloud-language transformers
Give them all at once to pip.
pip install google-cloud-texttospeech attrdict google-cloud-language transformers
It will try all the combinations of versions and dependencies' versions until it finds something suitable. This will potentially download a ton of packages just to see their dependencies, so you only want to make this once.
If happy with the result, extract the requirements file.
pip freeze > requirements.txt
This contains all the packages installed, we are not interested in all.
And from it, extract the specific versions of your desired packages.
cat requirements.txt | egrep -i "google-cloud-texttospeech|attrdict|google-cloud-language|transformers"
Note: The command above may not list all your required packages, just the ones that appear in the requirements.txt, so check that all show up.
attrdict==2.0.1
google-cloud-language==1.2.0
google-cloud-texttospeech==2.12.3
transformers==2.11.0
Now you can put that on a file like resolved-dependencies.txt
And next time, install the packages directly with the valid & compatible version with.
pip install -r resolved-dependencies.txt
pip install -e . is a great feature. It allows you to work on your package without having to uninstall-reinstall constantly. It seemingly doesn't, however, keep track of your build files (e.g. your setup.cfg or setup.py). Say, you change these (e.g. add, subtract, change version for dependencies, or change which modules you include in the package). What is then the best way to have pip recheck these requirements?
Or more generally, what is the way you're supposed to handle changes in your setup.cfg or setup.py when using pip install -e .?
What I usually end up doing is just doing pip install -e . in the root directory. This walks through your entire setup configuration again, install any new or changed dependencies and then uninstalls your package before reinstalling it again. Which definitely isn't always necessary, and does slow things down a lot.
While this certainly works, it feels against the idea of the 'editable' package.
Is there a proper way of doing this?
FYI, I know you can just install dependencies that aren't listed in your setup.cfg yourself by just pip install ..., my question is aimed at learning a better way of doing things.
It is probably the best way of doing things. The reason pip install -e . reinstalls everything is because it is environment agnostic.
Let's say you have two dependencies numpy >= 1.7.2 and pandas==1.4.2.
Now, pandas 1.4.2 requires a minimum version of numpy==1.18.2, so when you do a pip install (editable or not), it would probably pick the greatest compatible version(1.22.3).
Let's say you now want to fix numpy at 1.20.2. The only way pip knows it is compatible with pandas is by walking through your whole list of requirements.
If you do end up wanting a "better" tool than pip, check out pipenv or poetry.
How to list in Python project all libraries and their version?
(similar to Java maven pom.xml and Node.js npm 'package.json')
There is definitely way to specify that on command line,
e.g. pip install pandas==0.23, but that only hopefully will be listed in README, but most likely be forgotten.
Also as there is pip and conda, do they address this? Or maybe some other 3rd tool tries to unify for whatever pip or conda usage?
There are two main mechanisms to specify libraries for a package/repository:
For an out-of-the-box package that can be installed with all dependencies, use setup.py/pyproject.toml. These are used by package managers to automatically install required dependencies on package installation.
For a reproducible installation of a given setup of possibly many packages, use requirements.txt. These are used by package managers to explicitly install required dependencies.
Notably, the two handle different use-cases: Package dependencies define the general dependencies, only restricting dependency versions to specific ranges when needed for compatibility. Requirements define the precise dependencies, restricting all direct and indirect dependency versions to replicate some pre-existing setup.
I have developers that want to be able to publish a library as a "beta" or "release" version.
i.e.:
1.2.3-beta
1.2.3
On the consuming project, they couldn't give me any specific criteria of when they would want to use a beta or release package. We have CI in place, but without any definitive "when" I can't support two separate pip feeds, as they may flip flop. So I've suggested taking advantage of version range syntax in the requirements file, so they can just specify during checkin what they want. They've never had to do anything like this, and I'm basically a python rookie. Is it possible to filter on pre-release labels? I.e.
Will lib == 1.*.*-* pick up a beta package?
and
Will lib == 1.*.*, !=1.*.*-* pick up a release package and be sure to exclude any beta packages?
I would try out my theory myself, but I don't know python well enough to mock up some sort of sample libs locally, and they're too busy to research it.
By default pip will not install prereleases, like 1.0.0.b1
To enable installation of prereleases, you use the --pre flag with pip.
You can also use prerelease version specifiers to force pip to consider prereleases for individual packages without needing to use --pre. From https://pip.pypa.io/en/stable/reference/pip_install/#pre-release-versions:
If a Requirement specifier includes a pre-release or development version (e.g. >=0.0.dev0) then pip will allow pre-release and development versions for that requirement.
So in your requirements.txt file, you'd have something like:
package_a>=1.2.3 # will not install pre-releases
package_b>=1.2.3.dev0 # will install pre-releaes