I created a python 3.3 app on RedHat's Openshift cloud service. By default it has setup.py for my project. I'm learning Udemy course called "Build a SaaS app with Flask" (source code) Now I wanted to use python-click, as recommended by the course. It needs another setup.py for cli project; so to put that file in the project root folder, I renamed it to setup_cli.py. Now there are two files: setup.py and setup_cli.py.
Pip install seems to automatically look into setup.py.
# See Dockerfile in github source
pip install --editable <from setup_cli.py>
Can pip install --editable be used to point to setup_cli.py?
It seems that you can't do anything about it :-) - It's hard coded in pip source code :-)
If you try to use pip install -e ., it will call a method named parse_editable which will run this line:
if not os.path.exists(os.path.join(url_no_extras, 'setup.py')):
raise InstallationError(
"Directory %r is not installable. File 'setup.py' not found." %
url_no_extras
)
You may want to to use this command pip install -e file:///full/path/to/setup_cli.py, but this command also appends a hard coded setup.py to your path :-)
In setup_py there is this line:
setup_py = os.path.join(self.setup_py_dir, 'setup.py')
so as #cel commented, it seems that python <whatever-setup.py> develop is your only option.
Related
I am developing a python package, and I don't want to have to keep running pip install . to reinstall my package every time I change something. Using the -e or --editable doesn't seem to work unless I have a setup.py file, nor does --no-use-pep517. I have a pyproject.toml instead, as is preferred nowadays if I am not mistaken. So, what is the preferred way to do this nowadays?
My package is just a CLI script, but it imports some functions from another file in the same directory called utils.py. When developing, I can't just run the script manually rfrom the terminal, because then I get name_of_package is not a package from the line
from name_of_package.utils import function, whereas If i just have
from utils import function, I can run the script from the terminal, but when I pip install it, it says there is no module named utils.
I did install poetry and installed my dependencies, ran poetry shell and then tried to run my script with poetry run /path/to/script.py, but I kept getting an error that my package wasn't a package.
If you want to keep using setuptools as "build back-end" then you can replace the setup.py script with a setup.cfg declarative configuration file and still be able to do "editable" installations (independently of whether or not you have a pyproject.toml file).
There is now PEP 660 which standardizes editable installations. The following tools have support for PEP 660:
PDM
Flit
Hatch
Poetry
On top of setuptools-based projects, all projects that use a PEP 660 build back-end should be installable as editable by pip (python -m pip install --editable .).
I'm new to python, and I was wondering if you could help me run a python script. I'm trying to run a script called PunchBox from Github: https://github.com/psav/punchbox. So far, I have Python 3.9.5 and Git Bash.
In the GitHub page, it says:
To install, clone the repo, cd into it and then execute the following:
virtualenv -p python2 .pb2
source .pb2/bin/activate
pip install -U pip
pip install .
What does this mean exactly? Where do I run this code?
So far, I tried downloading the zip file from GitHub, installing Python 3.5.9, using cmd, finding the directory with cd, and running that code; but got an error:
Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name punchbox was given, but was not able to be found.
error in punchbox setup command: Error parsing C:\Users\Mi\Downloads\punchbox-master\punchbox-master\setup.cfg: Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name punchbox was given, but was not able to be found.
There's also a requirements.txt that lists additional scripts needed:
pre-commit
click
mido
pbr
PyYAML
svgwrite
Do these install automatically upon running the script for the first time?
I'm a little confused why I'm getting an error. Do you know what I'm doing wrong?
Thank you so much!
Giovanni
I assume you are new to programming. You have to write these lines in a terminal.
On Windows, it is Command Prompt or PowerShell Applications (latter preferred). On macOS, it is terminal
Copy all these lines at once, and paste them to your preferred terminal. The terminal will automatically run these one after the another.
FYI: Venv is a python package to create a virtual environment. The preceding commands set up the environment. Now install the required dependencies using this command instead of the last command (pip install .)
pip install -r requirements.txt
Based on your comment, it looks like you don't have virtualenv installed in your system. You may install it using the command pip install virtualenv.
Now, as you are using a Windows machine, you may open a Command Prompt or Windows PowerShell window and navigate to the directory where your cloned project resides.
Now, execute the following commands.
virtualenv -p python2 .pb2
.pb2\Scripts\activate.bat
pip install -U pip
pip install -r requirements.txt
Once you are done working in your virtual environment (which is named .pb2), you may close it by executing deactivate command.
#Giovanni T.
See, as far as you have installed Python and also downloaded the GitHub Repository as a zip file.
pip install -r requirements.txt
Just run this command.
Please make sure that the directory is pointing to the folder where this requirements.txt file is stored.
I don't understand this...
I want to install this https://gist.github.com/sixtenbe/1178136.
It is a peak detection script for python.
Everywhere I look I am told to use pip with the .git extension.
All I see is how to download the .zip, but from there I am lost.
How can I install this?
Thanks.
You can get the individual files in the Gist (or download the Gist as an ZIP and extract) and put them in your source code folder.
Then you will be able to import them as modules in your own scripts:
import analytic_wfm as AW
AW.ACV_A6( ... )
import peakdetect as PK
PK.peakdetect_parabola( ... )
Let's give it another look.
By "installing a package" we might mean that the package should be available via import.
For that the package directory should reside either in the current directory or in one of the other directories in the import search path.
One such directory is the "user-specific site-packages directory, USER_SITE":
python -c "import site; print(site.getusersitepackages())"
Git URL
First we might need a Git URL. Going to https://gist.github.com/sixtenbe/1178136 we can click on the Embed pop-up and switch it to Clone via HTTPS:
in order to obtain the GIT URL: https://gist.github.com/1178136.git.
git and bash
Having the Git URL and the Unix shell (bash) we can install the package manually into the USER_SITE.
Let's go into the USER_SITE first:
cd $(python -c "import site; print(site.getusersitepackages())")
pwd
Now that we are in the USER_SITE, let's download the Gist:
git clone https://gist.github.com/1178136.git analytic_wfm
Finally, let's verify that the package is now available:
cd && python -c "import analytic_wfm.analytic_wfm; print(analytic_wfm.analytic_wfm.__all__)"
If numpy is installed, it prints
['ACV_A1', 'ACV_A2', 'ACV_A3', 'ACV_A4', 'ACV_A5', 'ACV_A6', 'ACV_A7', 'ACV_A8']
pip
Let's try to install a Gist package with pip.
For pip install we should prefix the Git URL with git+:
pip install --user git+https://gist.github.com/1178136.git
This gives us the error:
ERROR: git+https://gist.github.com/1178136.git does not appear to be a
Python project: neither 'setup.py' nor 'pyproject.toml' found.
Looks like the package we've picked is missing the necessary pip configuration!
Let's try another one:
pip install --user git+https://gist.github.com/bf91613a021a536c7ce16cdba9168604.git
Installs NP:
Successfully built llog
Installing collected packages: llog
Successfully installed llog-1.0
Particularly because it has the setup.py.
Note also that Gist does not support subfolders, and pip seems to depend on them in handling the packages argument, but the code in setup.py can workaround this by creating the package subfolder on the fly and copying the Python files there!
Hence if you want to import that Gist, https://gist.github.com/sixtenbe/1178136, with the rest of the requirements.txt dependencies, - you can fork it and add setup.py to the effect.
pypi
Given that the analytic-wfm can also be found at the Python Package Index, https://pypi.org/project/analytic-wfm/, you can install it with
pip install analytic-wfm
When would the -e, or --editable option be useful with pip install?
For some projects the last line in requirements.txt is -e .. What does it do exactly?
As the man page says it:
-e,--editable <path/url>
Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.
So you would use this when trying to install a package locally, most often in the case when you are developing it on your system. It will just link the package to the original location, basically meaning any changes to the original package would reflect directly in your environment.
Some nuggets around the same here and here.
An example run can be:
pip install -e .
or
pip install -e ~/ultimate-utils/ultimate-utils-proj-src/
note the second is the full path to where the setup.py would be at.
Concrete example of using --editable in development
If you play with this test package as in:
cd ~
git clone https://github.com/cirosantilli/vcdvcd
cd vcdvcd
git checkout 5dd4205c37ed0244ecaf443d8106fadb2f9cfbb8
python -m pip install --editable . --user
it outputs:
Obtaining file:///home/ciro/bak/git/vcdvcd
Installing collected packages: vcdvcd
Attempting uninstall: vcdvcd
Found existing installation: vcdvcd 1.0.6
Can't uninstall 'vcdvcd'. No files were found to uninstall.
Running setup.py develop for vcdvcd
Successfully installed vcdvcd-1.0.6
The Can't uninstall 'vcdvcd' is normal: it tried to uninstall any existing vcdvcd to then replace them with the "symlink-like mechanism" that is produced in the following steps, but failed because there were no previous installations.
Then it generates a file:
~/.local/lib/python3.8/site-packages/vcdvcd.egg-link
which contains:
/home/ciro/vcdvcd
.
and acts as a "symlink" to the Python interpreter.
So now, if I make any changes to the git source code under /home/ciro/vcdvcd, it reflects automatically on importers who can from any directory do:
python -c 'import vcdvcd'
Note however that at my pip version at least, binary files installed with --editable, such as the vcdcat script provided by that package via scripts= on setup.py, do not get symlinked, just copied to:
~/.local/bin/vcdcat
just like for regular installs, and therefore updates to the git repository won't directly affect them.
By comparison, a regular non --editable install from the git source:
python -m pip uninstall vcdvcd
python -m pip install --user .
produces a copy of the installed files under:
~/.local/lib/python3.8/site-packages/vcdvcd
Uninstall of an editable package as done above requires a new enough pip as mentioned at: How to uninstall editable packages with pip (installed with -e)
Tested in Python 3.8, pip 20.0.2, Ubuntu 20.04.
Recommendation: develop directly in-tree whenever possible
The editable setup is useful when you are testing your patch to a package through another project.
If however you can fully test your change in-tree, just do that instead of generating an editable install which is more complex.
E.g., the vcdvcd package above is setup in a way that you can just cd into the source and do ./vcdcat without pip installing the package itself (in general, you might need to install dependencies from requirements.txt though), and the import vcdvcd that that executable does (or possibly your own custom test) just finds the package correctly in the same directory it lives in.
From Working in "development" mode:
Although not required, it’s common to locally install your project in
“editable” or “develop” mode while you’re working on it. This allows
your project to be both installed and editable in project form.
Assuming you’re in the root of your project directory, then run:
pip install -e .
Although somewhat cryptic, -e is short for
--editable, and . refers to the current working directory, so together, it means to install the current directory (i.e. your
project) in editable mode.
Some additional insights into the internals of setuptools and distutils from “Development Mode”:
Under normal circumstances, the distutils assume that you are going to
build a distribution of your project, not use it in its “raw” or
“unbuilt” form. If you were to use the distutils that way, you would
have to rebuild and reinstall your project every time you made a
change to it during development.
Another problem that sometimes comes up with the distutils is that you
may need to do development on two related projects at the same time.
You may need to put both projects’ packages in the same directory to
run them, but need to keep them separate for revision control
purposes. How can you do this?
Setuptools allows you to deploy your projects for use in a common
directory or staging area, but without copying any files. Thus, you
can edit each project’s code in its checkout directory, and only need
to run build commands when you change a project’s C extensions or
similarly compiled files. You can even deploy a project into another
project’s checkout directory, if that’s your preferred way of working
(as opposed to using a common independent staging area or the
site-packages directory).
To do this, use the setup.py develop command. It works very similarly
to setup.py install, except that it doesn’t actually install anything.
Instead, it creates a special .egg-link file in the deployment
directory, that links to your project’s source code. And, if your
deployment directory is Python’s site-packages directory, it will also
update the easy-install.pth file to include your project’s source
code, thereby making it available on sys.path for all programs using
that Python installation.
It is important to note that pip uninstall can not uninstall a module that has been installed with pip install -e. So if you go down this route, be prepared for things to get very messy if you ever need to uninstall. A partial solution is to (1) reinstall, keeping a record of files created, as in sudo python3 -m setup.py install --record installed_files.txt, and then (2) manually delete all the files listed, as in e.g. sudo rm -r /usr/local/lib/python3.7/dist-packages/tdc7201-0.1a2-py3.7.egg/ (for release 0.1a2 of module tdc7201). This does not 100% clean everything up however; even after you've done it, importing the (removed!) local library may succeed, and attempting to install the same version from a remote server may fail to do anything (because it thinks your (deleted!) local version is already up to date).
As suggested in previous answers, there is no symlinks that are getting created.
How does '-e' option work? -> It just updates the file "PYTHONDIR/site-packages/easy-install.pth" with the project path specified in the 'command pip install -e'.
So each time python search for a package it will check this directory as well => any changes to the files in this directory is instantly reflected.
I have a git repository with many folders, one of them being a python module installable with pip, like this:
repo.git/
repo.git/folder1/
repo.git/folder2/
repo.git/mymodule/
repo.git/mymodule/__init__.py
repo.git/mymodule/setup.py
repo.git/mymodule/...
Right now I have to do the following to install:
git clone http://server/repo.git
cd repo
pip install mymodule
cd ..
rm -rf repo
Is it possible to install the module directly with pip without explicitly cloning ?
I tried:
pip install git+https://server/repo.git/mymodule/
pip install git+https://server/repo.git:mymodule/
But I get:
IOError: [Errno 2] No such file or directory: '/tmp/pip-88tlLm-build/setup.py'
There is a pull request regarding this feature, and it seems to have been merged to develop branch a month ago. The syntax is the following:
pip install -e git+https://git.repo/some_repo.git#egg=version_subpkg&subdirectory=repo # install a python package from a repo subdirectory
We probably have to wait for a while until it gets merged to master and is distributed.
UPDATE: This is now available and documented at https://pip.pypa.io/en/stable/cli/pip_install/#vcs-support as follows:
For projects where setup.py is not in the root of project,
"subdirectory" component is used. Value of "subdirectory" component
should be a path starting from root of the project to where setup.py
is located.
So if your repository layout is:
- pkg_dir/
- setup.py # setup.py for package ``pkg``
- some_module.py
- other_dir/
- some_file
- some_other_file
You'll need to use
pip install -e vcs+protocol://repo_url/#egg=pkg&subdirectory=pkg_dir
Note: On Windows, you must place the URL in double quotes, or you'll get an error "'subdirectory' is not recognized as an internal or external command". E.g., use:
pip install -e "vcs+protocol://repo_url#egg=pkg&subdirectory=pkg_dir"
It's been already stated in one of the comments under the correct answer, but just to highlight this issue: when executing this from Linux command line, you must escape the &-character since ampersand is telling the command line to run a command in background:
git+https://git.repo/some_repo.git#egg=version_subpkg\&subdirectory=repo
Notice the backslash before the ampersand. The escaping behaviour might depend on the Linux distro; I'm not an expert.
If you ignore this, you might run into a cryptic error like the following:
bash: (...) command not found