I'm developing a Flask application using Babel. Thanks to Distutils/Setuptools Integration, all the parameters of compile/extract/... functions are stored in setup.cfg and compiling the i18n files is as easy as
./setup.py compile_catalog
Great. Now I would like this to be done automatically when running
./setup.py install
In make's words, that would be letting install target depend on compile_catalog target.
The context
We store only translation (.po) files in the code repository. .gitignore excludes .mo and .pot files from being tracked.
When a developer pulls a new revision of the code, he runs
pip install -r requirements.txt
to update dependencies and install the project in development mode. Then, using above command line, he compiles the translation binary (.mo) files.
Is there a simple and recommended way to modify setup.py to do both operations in one step? Or am I trying to misuse setuptools?
Using a script that like this would work for development purposes:
#!/bin/sh
./setup.py compile_catalog
pip install -r requirements.txt
but I would like a solution that also works when the package is installed with usual setup.py install instructions, like if installed from PyPi.
Should I understand that setuptools are not meant to be used like this, and people distributing software compile their translation files either manually or using custom scripts when creating their archives, rather than relying on setup.py to compile them at installation time?
I didn't find many posts on the Internet addressing this. The ones I found involved running pybabel command line interface from a function in setup.py, which sounds like a shame as it misses the point of setuptools integration.
I think your demand is totally valid and I'm quite surprised that there seems to be no official guideline on how to accomplish this.
The project I working on now also went multilingual and this is what I did:
In setup.cfg, make appropriate entries so that compile_catalog can be run without options.
In setup.py, subclass the install command from setuptools:
setup.py:
from setuptools import setup
from setuptools.command.install import install
class InstallWithCompile(install):
def run(self):
from babel.messages.frontend import compile_catalog
compiler = compile_catalog(self.distribution)
option_dict = self.distribution.get_option_dict('compile_catalog')
compiler.domain = [option_dict['domain'][1]]
compiler.directory = option_dict['directory'][1]
compiler.run()
super().run()
Then, when calling setup(), register our InstallWithCompile command with the name "install" and make sure that the *.mo files will be included in the package:
setup(
...
cmdclass={
'install': InstallWithCompile,
},
...
package_data={'': ['locale/*/*/*.mo', 'locale/*/*/*.po']},
)
Since babel is used during the setup, you should add it as a setup dependency:
setup_requires=[
'babel',
],
Note that a package (here, babel) appearing in both setup_requires and install_requires won't be installed correctly using python setup.py install due to an issue in setuptools, but it works fine with pip install.
Related
I'm developing a python module with a long pybind11 C++ extension compilation step invoked via cmake in setup.py.
When making changes to a C++ file, cmake invoked via python setup.py develop will avoid recompiling units whose dependent files have not changed. However, invoking setup.py directly ignores the settings in my pyproject.toml and I understand that the modern way to do a developmental build is with python -m pip install -e .
While pip install -e successfully builds, it unfortunately starts from scratch inside a clean temporary directory every invocation. Is there a way to instruct pip install -e to maintain my CMakeCache.txt and compilation dependency tracking?
(And/or does this somehow indicate that I misunderstand pip install -e or am using it incorrectly?)
This previous unanswered question is quite similar sounding. Perhaps, I have the added detail that my python setup.py develop is working in this regard.
I'm moving on from single scripts to a bigger python application.
It's an application with multiple packages.
package1-> package1/.py files
package2-> package2/.py files
As package 1 should be able to be used stand alone, I keep it in a separate git repo.
I'd love to do in package2: import package1
It feels like the easiest way to do it would be having project1 (in its git repo) in a subdirectory of project2, but that doesnt sound like a nice solution.
Some answers I found feels dated and I couldn't get it to work. (python setup.py install)
Adding package1 location to the PATH is a solution, but it's not very nice if I want to distribute it to co-workers. Ideally, I "install" the package as easily as possible.
I read "pip" would be preferred, but would need some directions where to start looking for creating a package. Also, distribution would be only local.
(python3.6. Code will be used on linux and windows. )
excerpt from an excellent (but kindof hidden) answer using pip given by np8 in question
Importing modules from parent folder:
checkout his answer!
--
1) Add a setup.py to the root folder
The contents of the setup.py can be simply
from setuptools import setup, find_packages
setup(name='myproject', version='1.0', packages=find_packages())
Basically "any" setup.py would work. This is just a minimal working example.
2) Use a virtual environment
3) pip install your project in editable state
Install your top level package myproject using pip. The trick is to use the -e flag when doing the install. This way it is installed in an editable state, and all the edits made to the .py files will be automatically included in the installed package.
In the root directory, run
pip install -e . (note the dot, it stands for "current directory")
You can also see that it is installed by using pip freeze
When would the -e, or --editable option be useful with pip install?
For some projects the last line in requirements.txt is -e .. What does it do exactly?
As the man page says it:
-e,--editable <path/url>
Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.
So you would use this when trying to install a package locally, most often in the case when you are developing it on your system. It will just link the package to the original location, basically meaning any changes to the original package would reflect directly in your environment.
Some nuggets around the same here and here.
An example run can be:
pip install -e .
or
pip install -e ~/ultimate-utils/ultimate-utils-proj-src/
note the second is the full path to where the setup.py would be at.
Concrete example of using --editable in development
If you play with this test package as in:
cd ~
git clone https://github.com/cirosantilli/vcdvcd
cd vcdvcd
git checkout 5dd4205c37ed0244ecaf443d8106fadb2f9cfbb8
python -m pip install --editable . --user
it outputs:
Obtaining file:///home/ciro/bak/git/vcdvcd
Installing collected packages: vcdvcd
Attempting uninstall: vcdvcd
Found existing installation: vcdvcd 1.0.6
Can't uninstall 'vcdvcd'. No files were found to uninstall.
Running setup.py develop for vcdvcd
Successfully installed vcdvcd-1.0.6
The Can't uninstall 'vcdvcd' is normal: it tried to uninstall any existing vcdvcd to then replace them with the "symlink-like mechanism" that is produced in the following steps, but failed because there were no previous installations.
Then it generates a file:
~/.local/lib/python3.8/site-packages/vcdvcd.egg-link
which contains:
/home/ciro/vcdvcd
.
and acts as a "symlink" to the Python interpreter.
So now, if I make any changes to the git source code under /home/ciro/vcdvcd, it reflects automatically on importers who can from any directory do:
python -c 'import vcdvcd'
Note however that at my pip version at least, binary files installed with --editable, such as the vcdcat script provided by that package via scripts= on setup.py, do not get symlinked, just copied to:
~/.local/bin/vcdcat
just like for regular installs, and therefore updates to the git repository won't directly affect them.
By comparison, a regular non --editable install from the git source:
python -m pip uninstall vcdvcd
python -m pip install --user .
produces a copy of the installed files under:
~/.local/lib/python3.8/site-packages/vcdvcd
Uninstall of an editable package as done above requires a new enough pip as mentioned at: How to uninstall editable packages with pip (installed with -e)
Tested in Python 3.8, pip 20.0.2, Ubuntu 20.04.
Recommendation: develop directly in-tree whenever possible
The editable setup is useful when you are testing your patch to a package through another project.
If however you can fully test your change in-tree, just do that instead of generating an editable install which is more complex.
E.g., the vcdvcd package above is setup in a way that you can just cd into the source and do ./vcdcat without pip installing the package itself (in general, you might need to install dependencies from requirements.txt though), and the import vcdvcd that that executable does (or possibly your own custom test) just finds the package correctly in the same directory it lives in.
From Working in "development" mode:
Although not required, it’s common to locally install your project in
“editable” or “develop” mode while you’re working on it. This allows
your project to be both installed and editable in project form.
Assuming you’re in the root of your project directory, then run:
pip install -e .
Although somewhat cryptic, -e is short for
--editable, and . refers to the current working directory, so together, it means to install the current directory (i.e. your
project) in editable mode.
Some additional insights into the internals of setuptools and distutils from “Development Mode”:
Under normal circumstances, the distutils assume that you are going to
build a distribution of your project, not use it in its “raw” or
“unbuilt” form. If you were to use the distutils that way, you would
have to rebuild and reinstall your project every time you made a
change to it during development.
Another problem that sometimes comes up with the distutils is that you
may need to do development on two related projects at the same time.
You may need to put both projects’ packages in the same directory to
run them, but need to keep them separate for revision control
purposes. How can you do this?
Setuptools allows you to deploy your projects for use in a common
directory or staging area, but without copying any files. Thus, you
can edit each project’s code in its checkout directory, and only need
to run build commands when you change a project’s C extensions or
similarly compiled files. You can even deploy a project into another
project’s checkout directory, if that’s your preferred way of working
(as opposed to using a common independent staging area or the
site-packages directory).
To do this, use the setup.py develop command. It works very similarly
to setup.py install, except that it doesn’t actually install anything.
Instead, it creates a special .egg-link file in the deployment
directory, that links to your project’s source code. And, if your
deployment directory is Python’s site-packages directory, it will also
update the easy-install.pth file to include your project’s source
code, thereby making it available on sys.path for all programs using
that Python installation.
It is important to note that pip uninstall can not uninstall a module that has been installed with pip install -e. So if you go down this route, be prepared for things to get very messy if you ever need to uninstall. A partial solution is to (1) reinstall, keeping a record of files created, as in sudo python3 -m setup.py install --record installed_files.txt, and then (2) manually delete all the files listed, as in e.g. sudo rm -r /usr/local/lib/python3.7/dist-packages/tdc7201-0.1a2-py3.7.egg/ (for release 0.1a2 of module tdc7201). This does not 100% clean everything up however; even after you've done it, importing the (removed!) local library may succeed, and attempting to install the same version from a remote server may fail to do anything (because it thinks your (deleted!) local version is already up to date).
As suggested in previous answers, there is no symlinks that are getting created.
How does '-e' option work? -> It just updates the file "PYTHONDIR/site-packages/easy-install.pth" with the project path specified in the 'command pip install -e'.
So each time python search for a package it will check this directory as well => any changes to the files in this directory is instantly reflected.
What are the differences between the below commands
python setup.py install develop
Doesn't work for me error No such file or directory: 'build/bdist.macosx-10.7-intel/egg/test-easy-install-37886.pth'
python setup.py develop
Works for me appears to make an .egg link file
python setup.py install
Works for me appears to make a .egg file which in .zip file format
Develop is a setuptools / distribute feature that allows you to add a project
to your Python environment without installing it - so you can continue
its "development"
In other words, when you call "python setup.py develop", setuptools will
compile the metadata and hook your project into Python's site-package,
but the packages and modules that will be used are the one in the
directory where you've run that command.
This is useful to continue working on your code and testing it without
having to run "python setup.py install" on every run
With develop, Python 'pseudo-installs' a package by running the setup.py script instead of install. The difference is a modification of the environment (it doesn't with develop), so a package can be imported from it's current location instead of a site-package directory. The advantage of this is you can develop packages that are being used by other packages, and you can modify source code in place with develop.
As far as "setup.py install develop", I've never seen anyone use that before, sorry.
source
source
source
python setup.py install develop
Is a wrong command.
When you use develop you use the current code when you run your application.
When you use install and then modify you code, your modifications will not be taken in account while running your app. until you rerun install or develop.
In a crusade to make my application pip-installable, I'm fighting big fights with setuptools and distribute. I assume my dependencies are correct, i.e. installing with pip install myapp should probably fill the virtual environment correctly. However, I'd like to streamline development while I'm at it, so my goal is to start with an empty virtualenv and make setup.py test (and later setup.py develop, but that's a whole different fight) fill it with all defined dependencies.
And now to my problem: no matter how hard I try, all I get are dependencies installed as .eggs in my project directory which is sub-optimal at the very least. I tried creating a new setuptools command which would use pip (which seems to work, even though awkwardly) but that can't seriously be the solution (subclassing and overriding that is).
So how do I make setup.py test fill the virtualevn instead of my working directory?
By design, you can't make the tests_requires or the setup_requires entries go into the virtual environment. The idea is to separate what is required for performing tests/setup and what is required to actually use the package being installed. For example, I may require that the "coverage" module be needed for running tests on my package, but it isn't used by any of my code in the package. Therefore, if I didn't have "coverage" in my environment when I go and run tests, I wouldn't want "coverage" to get installed into the environment if my package didn't need it.
If you are using setuptools, you can specify test dependencies using the tests_require keyword argument for the setup method.
from setuptools import setup
setup(
name='your-package-name',
version='1.0.0',
author='me',
author_email='me#mybasement.com',
install_requires=['Pygments>=1.4'],
tests_require=['nose'],
packages=[
'your_package_name',
],
)
When you run python setup.py test, this will check for nose and install it to the currently active virtualenv using pip if not already available.
Note that this argument will be ignored if you are using distribute.core.setup (nor will a test command be available).