I created a small Python package. As I'm making modifications, I've found that I need to do the following in order for the changes to take effect:
cd to the package's root directory (same as where setup.py is located)
pip uninstall <package_name>
delete the build directory that was created when the package was last installed
python setup.py install
The key is step #3. I've noticed that the changes don't stick if the build directory isn't removed first. However, I don't know why this step is necessary and I can't find any references to having to do this and why. This is true for both Windows and Mac OSX. Can somebody please provide a rationale or point me in the right direction?
python setup.py install is implicitly equivalent to two subcommands:
build, which puts the files to install into the build directory, and
install, which copies files from the build/lib directory into the installation directory.
It appears that the first step is only performed when necessary, i.e. when you run python setup.py install without first removing the build directory, you are installing stale modules from the previous build.
To have changes take effect immediately while developing, you should run python setup.py develop, which creates a link in the installation directory back to the source code.
Related
On Linux, every time I try to install a python package I get an error that the site-packages directory is missing. For example:
error: [Errno 2] No such file or directory: '/opt/python/cp39-cp39/Lib/site-packages/'
One simple solution is to just create the missing directory; however, this is not possible in certain instances. For example, when running python -m build, a venv with a random name is created to build the package in - therefore, I cannot create the site-package folder since there is no pause where I can modify the new venv.
I have tested this on multiple machines, and this issue also appears in GitHub hosted CI/CD runners. This is especially difficult to resolve in CI/CD pipelines because I have no control over additional venvs created by build, cibuildwheel and other PyPA tools.
This error occurs when I run pip install ., python -m build and several other pip commands that involve building wheel packages. The error always occurs while pip is building the wheel.
I would appreciate any suggestions!
When would the -e, or --editable option be useful with pip install?
For some projects the last line in requirements.txt is -e .. What does it do exactly?
As the man page says it:
-e,--editable <path/url>
Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.
So you would use this when trying to install a package locally, most often in the case when you are developing it on your system. It will just link the package to the original location, basically meaning any changes to the original package would reflect directly in your environment.
Some nuggets around the same here and here.
An example run can be:
pip install -e .
or
pip install -e ~/ultimate-utils/ultimate-utils-proj-src/
note the second is the full path to where the setup.py would be at.
Concrete example of using --editable in development
If you play with this test package as in:
cd ~
git clone https://github.com/cirosantilli/vcdvcd
cd vcdvcd
git checkout 5dd4205c37ed0244ecaf443d8106fadb2f9cfbb8
python -m pip install --editable . --user
it outputs:
Obtaining file:///home/ciro/bak/git/vcdvcd
Installing collected packages: vcdvcd
Attempting uninstall: vcdvcd
Found existing installation: vcdvcd 1.0.6
Can't uninstall 'vcdvcd'. No files were found to uninstall.
Running setup.py develop for vcdvcd
Successfully installed vcdvcd-1.0.6
The Can't uninstall 'vcdvcd' is normal: it tried to uninstall any existing vcdvcd to then replace them with the "symlink-like mechanism" that is produced in the following steps, but failed because there were no previous installations.
Then it generates a file:
~/.local/lib/python3.8/site-packages/vcdvcd.egg-link
which contains:
/home/ciro/vcdvcd
.
and acts as a "symlink" to the Python interpreter.
So now, if I make any changes to the git source code under /home/ciro/vcdvcd, it reflects automatically on importers who can from any directory do:
python -c 'import vcdvcd'
Note however that at my pip version at least, binary files installed with --editable, such as the vcdcat script provided by that package via scripts= on setup.py, do not get symlinked, just copied to:
~/.local/bin/vcdcat
just like for regular installs, and therefore updates to the git repository won't directly affect them.
By comparison, a regular non --editable install from the git source:
python -m pip uninstall vcdvcd
python -m pip install --user .
produces a copy of the installed files under:
~/.local/lib/python3.8/site-packages/vcdvcd
Uninstall of an editable package as done above requires a new enough pip as mentioned at: How to uninstall editable packages with pip (installed with -e)
Tested in Python 3.8, pip 20.0.2, Ubuntu 20.04.
Recommendation: develop directly in-tree whenever possible
The editable setup is useful when you are testing your patch to a package through another project.
If however you can fully test your change in-tree, just do that instead of generating an editable install which is more complex.
E.g., the vcdvcd package above is setup in a way that you can just cd into the source and do ./vcdcat without pip installing the package itself (in general, you might need to install dependencies from requirements.txt though), and the import vcdvcd that that executable does (or possibly your own custom test) just finds the package correctly in the same directory it lives in.
From Working in "development" mode:
Although not required, it’s common to locally install your project in
“editable” or “develop” mode while you’re working on it. This allows
your project to be both installed and editable in project form.
Assuming you’re in the root of your project directory, then run:
pip install -e .
Although somewhat cryptic, -e is short for
--editable, and . refers to the current working directory, so together, it means to install the current directory (i.e. your
project) in editable mode.
Some additional insights into the internals of setuptools and distutils from “Development Mode”:
Under normal circumstances, the distutils assume that you are going to
build a distribution of your project, not use it in its “raw” or
“unbuilt” form. If you were to use the distutils that way, you would
have to rebuild and reinstall your project every time you made a
change to it during development.
Another problem that sometimes comes up with the distutils is that you
may need to do development on two related projects at the same time.
You may need to put both projects’ packages in the same directory to
run them, but need to keep them separate for revision control
purposes. How can you do this?
Setuptools allows you to deploy your projects for use in a common
directory or staging area, but without copying any files. Thus, you
can edit each project’s code in its checkout directory, and only need
to run build commands when you change a project’s C extensions or
similarly compiled files. You can even deploy a project into another
project’s checkout directory, if that’s your preferred way of working
(as opposed to using a common independent staging area or the
site-packages directory).
To do this, use the setup.py develop command. It works very similarly
to setup.py install, except that it doesn’t actually install anything.
Instead, it creates a special .egg-link file in the deployment
directory, that links to your project’s source code. And, if your
deployment directory is Python’s site-packages directory, it will also
update the easy-install.pth file to include your project’s source
code, thereby making it available on sys.path for all programs using
that Python installation.
It is important to note that pip uninstall can not uninstall a module that has been installed with pip install -e. So if you go down this route, be prepared for things to get very messy if you ever need to uninstall. A partial solution is to (1) reinstall, keeping a record of files created, as in sudo python3 -m setup.py install --record installed_files.txt, and then (2) manually delete all the files listed, as in e.g. sudo rm -r /usr/local/lib/python3.7/dist-packages/tdc7201-0.1a2-py3.7.egg/ (for release 0.1a2 of module tdc7201). This does not 100% clean everything up however; even after you've done it, importing the (removed!) local library may succeed, and attempting to install the same version from a remote server may fail to do anything (because it thinks your (deleted!) local version is already up to date).
As suggested in previous answers, there is no symlinks that are getting created.
How does '-e' option work? -> It just updates the file "PYTHONDIR/site-packages/easy-install.pth" with the project path specified in the 'command pip install -e'.
So each time python search for a package it will check this directory as well => any changes to the files in this directory is instantly reflected.
What are the differences between the below commands
python setup.py install develop
Doesn't work for me error No such file or directory: 'build/bdist.macosx-10.7-intel/egg/test-easy-install-37886.pth'
python setup.py develop
Works for me appears to make an .egg link file
python setup.py install
Works for me appears to make a .egg file which in .zip file format
Develop is a setuptools / distribute feature that allows you to add a project
to your Python environment without installing it - so you can continue
its "development"
In other words, when you call "python setup.py develop", setuptools will
compile the metadata and hook your project into Python's site-package,
but the packages and modules that will be used are the one in the
directory where you've run that command.
This is useful to continue working on your code and testing it without
having to run "python setup.py install" on every run
With develop, Python 'pseudo-installs' a package by running the setup.py script instead of install. The difference is a modification of the environment (it doesn't with develop), so a package can be imported from it's current location instead of a site-package directory. The advantage of this is you can develop packages that are being used by other packages, and you can modify source code in place with develop.
As far as "setup.py install develop", I've never seen anyone use that before, sorry.
source
source
source
python setup.py install develop
Is a wrong command.
When you use develop you use the current code when you run your application.
When you use install and then modify you code, your modifications will not be taken in account while running your app. until you rerun install or develop.
I use virtualenv to create isolated environments for my Python projects. Then i install dependencies with pip - Python package manager. Sometimes i forget to do source venv/bin/activate, and then pip creates build/ directories inside my projects. Why does pip create them? May i delete them, and if not, may i put them in my .hgignore file?
As far as i understand, pip stores source of downloaded packages there along a file called pip-delete-this-directory.txt. But when i delete it, everything still works, as the real code is being put into venv/lib/python2.7/site-packages/. Then what is build/ really for?
The build directory is where a packages gets unpacked into and build from. When the package is installed successfully, pip removes the unpacked dir from build, unless you've removed pip-delete-this-directory.txt. As described in pip-delete-this-directory.txt:
This file is placed here by pip to indicate the source was put
here by pip.
Once this package is successfully installed this source code will be
deleted (unless you remove this file).
Thus its less important for runtime environment. You could ignore it safely.
Also, you could use pip install -b customized_build_directory to specify another directory as build base directory, for example /tmp
Furthermore, you could pip install --no-download package_name to rebuild the package w/o downloading it, if the previous installation of the package failed.
I am having real trouble installing SUDS in python 2.6.4. I have tried to install the setup file but it says the location of python cannot be found. This is because I have changed the location of python. I have tried to use easy_install but am having no luck. Does anyone know a simple way to do this or have a link to clear installation instructions.
Command that I entered was:
python setup.py install
The result I recieved was:
running install
error: cannot create or remove files in install directory
The following error occurred while trying to add or remove files in the
installation directory:
[Errno 13] Permission denied: '/usr/local/lib/python2.6/site-packages/test-easy-install-9203.write-test'
The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/usr/local/lib/python2.6/site-packages/
Perhaps your account does not have write access to this directory? If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account. If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.
For information on other options, you may wish to consult the
documentation at:
http://peak.telecommunity.com/EasyInstall.html
And if I have to change the python path how exactly do you do this.
I have tried what one site said to do and it was to first, create an altinstall.pth file in Python's site-packages directory, containing the following line:
import os, site; site.addsitedir(os.path.expanduser('~/lib/python2.3'))
Then it says modify distutils.cfg in the distutils directory with:
[install]
install_lib = ~/lib/python2.3
# This next line is optional but often quite useful; it directs EasyInstall
# and the distutils to install scripts in the user's "bin" directory. For
# Mac OS X framework Python builds, you should use /usr/local/bin instead,
# because neither ~/bin nor the default script installation location are on
# the system PATH.
#
install_scripts = ~/bin
Have you tried setting PYTHONPATH to the location of python? Maybe this way it will know, where to install it.
You are calling it with python setup.py install. Try sudo python setup.py install, if you are using some linux and you are sudoer.
I got messages like this too when I installed suds and python-ntlm. Our site has a separate areafor installations so that we can maintain multiple versions, so my first installation step was
python setup.py install --prefix=/install/suds/suds-0.4
and I got the same messages about installplace. To fix:
Make sure the directories are there with
mkdir -p /install/suds/suds-0.4/lib/python2.6/site-packages/
(This surprised me a little, I thought setup would build the directories.)
Make sure you have write permission down the tree with
chmod -R 775 /install/suds/suds-0.4/lib/python2.6/site-packages/
Neither of which got rid of the message!
The last step was to put the install area into PYTHONPATH, and then do the setup.py
export PYTHONPATH=/install/suds/suds-0.4/lib/python2.6/site-packages:$PYTHONPATH
python setup.py install --prefix=/opt/sw/fw/qce/suds/suds-0.4
with a final chmod to make the newly installed files readable in case umask is set to something restrictive:
chmod 755 /install/suds/suds-0.4/lib/python2.6/site-packages/*
After this I could start python and import suds. The key step was the putting the suds site-packages directory into PYTHONPATH.
I expect this help comes too late to help the original poster, but I hope it helps someone else who come to SO with this question. As I did.
I would need more details of your OS to give a fully accurate response. From the sounds of your question, you changed your path of python. Normally you'll have a preinstalled version of python that is compatible with your OS. For example, CentOS 5.x comes with python 2.4, however you can do a yum install of python 2.6. Once installed, you can run python 2.6 by the python26 command.
When doing installs and packages, I would recommend that you try to use package managers as much as possible, as they help take care of your dependencies, such as yum. Yum also helps control updating packages instead of having to do updates manually. The next best thing is to do installs via pip or easy install, in the case of this question, you can try easy_install https://fedorahosted.org/releases/s/u/suds/python-suds-0.4.tar.gz (requires setuptools), and as a last resort, you can try to do the manual install. I if I get the point that I'm doing a manual install, I feel I failed somewhere :)
Others have given good detail on on how to do the install manually.
Good luck.