Question regarding installation of a Python-package from a private git-repository.
I have an init.py-file that is run whenever a user logs in to my service. This script is responsible for installing required packages, amongst others a python-package (with setup.py) from a private repository.
I am looking for ways to:
Install the latest version of the package if not currently installed.
Update the package, if the current installed version is not the latest.
Perform no action, if the latest version of the package already is installed.
I have tried the following:
Using pip install --upgrade git+ssh://..., however this always performs a clean install of the package.
Using pip install git+ssh://..., however this will not update the package if the current version is not the latest.
I am currently looking into ways of doing this manually by:
Git cloning the repository if it does not exist locally; then,
Call python setup.py develop to install the package in develop mode; then finally,
Do a git stash; git pull to discard any changes to working directory, and automatically pull latest changes.
However, I feel this approach is prone to users messing up.
I'd love if someone could provide some insight into this issue.
Thanks in advance!
Related
It's a great hassle when installing some packages in a VE and conda or pip downloads them again even when I already have it in my base environment. Since I have limited internet bandwidth and I'm assuming I'll work with many different VE's, it will take a lot of time to download basic packages such as OpenCV/Tensorflow.
By default, pip caches anything it downloads, and will used the cached version whenever possible. This cache is shared between your base environment and all virtual environments. So unless you pass the --no-cache-dir option, pip downloading a package means it has not previously downloaded a compatible version of that package. If you already have that package installed in your base environment or another virtual environment and it downloads it anyway, this probably means one or more of the following is true:
You installed your existing version with a method other than pip.
There is a newer version available, and you didn't specify, for example, pip install pandas=1.1.5 (if that's the version you already have elsewhere). Pip will install the newest compatible version for your environment, unless you tell it otherwise.
The VE you're installing to is a different Python version (e.g. created with Pyenv), and needs a different build.
I'm less familiar with the specifics of conda, and I can't seem to find anything in its online docs that focuses on the default caching behavior. However, a how-to for modifying the cache location seems to assume that the default behavior is similar to how pip works. Perhaps someone else with more Anaconda experience can chime in as well.
So except for the caveats above, as long as you're installing a package with the same method you did last time, you shouldn't have to download anything.
If you want to simplify the process of installing all the same packages (that were installed via pip) in a new VE that you already have in another environment, pip can automate that too. Run pip freeze > requirements.txt in the first environment, and copy the resulting file to your newly created VE. There, run pip install -r requirements.txt and pip will install all the packages that were installed (via pip) in the first environment. (Note that pip freeze records version numbers as well, so this won't install newer versions that may be available -- whether this is a good or bad thing depends on your needs.)
I wonder if it's possible to grab the change logs of updated packages after running
pip install [package_name] --upgrade
or
pipenv update
Most of packages will have change logs in their repo. e.g
https://github.com/kennethreitz/requests-html/releases
https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst
https://github.com/urllib3/urllib3/blob/master/CHANGES.rst
It will be more productive if I could have the latest updates via CLI.
As a work around, I have been using the Changelogs module it uses heuristics to attempt to get a changelog, but of course due to plenty of packages doing their own thing, it will never be able to work for all packages.
I did test it on your 3 sample packages, requests, pytest, urllib3 and was successful for printing out a changelog for them.
The question
Ansible is a python moduel, installable via pip. It relies on several dependencies, also pip modules. Is it possible to "roll up" all of those dependencies and Ansible itself into some sort of a single package, that can be installed offline, without root? It's highly preferable to not need pip for the install, although it will be available for package creation.
Extra background
I'm trying to install Ansible on one of our servers. The server does not have access to the internet, there is no root access. Pip is not installed, but Python is. It is possible to get pip installed there, but might be complicated. The only way to get anything on the server is via an internal tar.gz package sharing solution.
I've tried fiddling around with rpm, saving dependencies, but the absence of root access put an end to that.
Use pip on an internet-connected machine to download all the deps to a local dir with --download and -r requirements.txt, then drop that dir on the disconnected machine with pip installed, and install using --no-index and --find-links=(archive dir).
See https://pip.pypa.io/en/latest/user_guide/#fast-local-installs
I was having problems with one package not doing what I read in it's documentation, until I noticed that pip installed a outdated version.
On the pip package page it would seem like it was last update 2014, but when I installed, the package files were versioned mid 2013.
How does updating pip packages work and who should be doing it? The project maintainer (on github, or on pip pages?)?
All packages that can be downloaded with PIP are actually hosted on the Python Package Index. The Python organization collaborates with project maintainers to host the projects.
The problem with having outdated packages on pip that do not align with the documentation and current state on github can be really annoying. Despite that you did not ask for a workaround I would like to contribute one in case that other users might land on this page looking for such.
First uninstall the package you installed via pip before:
pip uninstall package
Next install the latest version directly from the github repo:
pip install git+https://github.com/user/package.git
The cool thing about this is that you can still manage your packages with pip but your not limited by what version is available on the Python Package Index.
The simplest way to deal with python package installations, so far, to me, has been to check out the source from the source control system and then add a symbolic link in the python dist-packages folder.
Clearly since source control provides the complete control to downgrade, upgrade to any branch, tag, it works very well.
Is there a way using one of the Package installers (easy_install or pip or other), one can achieve the same.
easy_install obtains the tar.gz and install them using the setup.py install which installs in the dist-packages folder in python2.6. Is there a way to configure it, or pip to use the source version control system (SVN/GIT/Hg/Bzr) instead.
Using pip this is quite easy. For instance:
pip install -e hg+http://bitbucket.org/andrewgodwin/south/#egg=South
Pip will automatically clone the source repo and run "setup.py develop" for you to install it into your environment (which hopefully is a virtualenv). Git, Subversion, Bazaar and Mercurial are all supported.
You can also then run "pip freeze" and it will output a list of your currently-installed packages with their exact versions (including, for develop-installs, the exact revision from the VCS). You can put this straight into a requirements file and later run
pip install -r requirements.txt
to install that same set of packages at the exact same versions.
If you download or check out the source distribution of a package — the one that has its "setup.py" inside of it — then if the package is based on the "setuptools" (which also power easy_install), you can move into that directory and say:
$ python setup.py develop
and it will create the right symlinks in dist-packages so that the .py files in the source distribution are the ones that get imported, rather than copies installed separately (which is what "setup.py install" would do — create separate copies that don't change immediately when you edit the source code to try a change).
As the other response indicates, you should try reading the "setuptools" documentation to learn more. "setup.py develop" is a really useful feature! Try using it in combination with a virtualenv, and you can "setup.py develop" painlessly and without messing up your system-wide Python with packages you are only developing on temporarily:
http://pypi.python.org/pypi/virtualenv
easy_install has support for downloading specific versions. For example:
easy_install python-dateutil==1.4.0
Will install v1.4, while the latest version 1.4.1 would be picked if no version was specified.
There is also support for svn checkouts, but using that doesn't give you much benefits from your manual version. See the manual for more information above.
Being able to switch to specific branches is rarely useful unless you are developing the packages in question, and then it's typically not a good idea to install them in site-packages anyway.
easy_install accepts a URL for the source tree too. Works at least when the sources are in Subversion.