I have a python package targeted at linux machines that needs to install its locale files to an accessible location. Right now, I have them being installed to sys.prefix + "/share/locale/".
However, I found a small caveat with Ubuntu and pip. Under default conditions, Ubuntu installs packages installed with pip to /usr/local and sets sys.prefix to that during installation. However, after installation, when the package is run, the prefix is /usr, meaning my code can't find the locale files installed at /usr/local.
I could simply hardcode the location, but I would prefer not to do this, as it makes the package less portable and would require the user to install it as root. These are added as data_files in setup.py and won't be discoverable as a python package.
How else can I ensure my package can find my the locale files after
installation?
I thought about adding a line to the package's __init__.py during installation, which created a variable pointing to the locale dir's location. However, it did not seem trivial to edit files being installed without changing the source files.
This is a python 3 only package.
Maybe use the resource functions available in pkg_resources to find the files?
from pkg_resources import resource_stream, resource_filename
with resource_stream('my_package', 'locale/foo.dat') as infp:
# ...
# ... or ...
foo_location = resource_filename('my_package', 'locale/foo.dat')
Related
Is there any way to check whether a Python package has been installed normally (pip install / setup.py install) or in editable/egg-link mode (pip install -e / setup.py develop)?
I know I could check whether the path to the package contains site-packages which would most likely mean it's a "non-editable" install, but this feels extremely dirty and I would rather avoid this.
The reason I'm trying to check this is that my application is checking for config files in various places, such as /etc/myapp.conf and ~/.myapp.conf. For developers I'd like to check in <pkgdir>/myapp.conf but since I show the list of possible locations in case no config was found, I really don't want to include the pkgdir option when the package has been installed to site-packages (since users should not create a config file in there).
pip contains code for this (it's used by pip freeze to prefix the line with -e). Since pip's API is not guaranteed to be stable, it's best to copy the code into the own application instead of importing it from pip:
def dist_is_editable(dist):
"""Is distribution an editable install?"""
for path_item in sys.path:
egg_link = os.path.join(path_item, dist.project_name + '.egg-link')
if os.path.isfile(egg_link):
return True
return False
The code is MIT-licensed so it should be safe to copy&paste into pretty much any project.
I have a very basic virtualenv setup on OSX with two packages, py3dns, and dnspython. Both packages have a dns module(actually one is DNS, the other dns, but on OSX they are the same), and these modules both have files named opcode.py.
Virtual env is installing both of these modules into lib/python3.5/site-packages/, and not into lib/python3.5/site-packages/#{package}, therefore these opcode.py files are overwriting each other.
Is this a bug(feature?) of virtualenv? Is there some way around this? Unfortunately I dont have the option of using only one of these dns packages, as they are both indirectly imported via other packages that I need to use in my project. Any advice would be appreciated.
You can specify different target path https://pip.pypa.io/en/latest/reference/pip_install/#cmdoption-t
You can install second package like this:
pip3 install dnspython --target /Users/green/dns
and then create file dns.pth in site-packages with path to package:
/Users/green/dns
and in your code:
from dns.opcode import from_text
from DNS.Opcode import opcodemap
print("ok")
I'm using computers running Debian or Ubuntu.
In the current configuration of my desktop computer (running Xubuntu 14.04), ${HOME}/.local/lib/python2.7/site-packages [^1] appears after /usr/lib/python2.7/dist-packages [^2] in sys.path.
It's the contrary in the other computers I use, and the latter is the desired behaviour for me: I want my locally installed packages to prevail over the system-wide installed ones.
How can I figure out where the configuration tweak in my desktop computer comes from?
Do I have to go through every *.pth file I find in the system and try to understand what it specifies?
[^1]: where the packages installed via python setup.py --user go
[^2]: where the package installed via apt-get install seem to go
Update:
My question seems related to How does python load all the values in sys.path?
This answer led me to try this strace trick and I could see that the first opened .pth file was /usr/local/lib/python2.7/dist-packages/easy-install.pth which contains the following code:
import sys; sys.__plen = len(sys.path)
./cutadapt-1.3-py2.7-linux-x86_64.egg
./DendroPy-3.12.0-py2.7.egg
./pylint-1.4.0-py2.7.egg
./astroid-1.3.2-py2.7.egg
/usr/lib/python2.7/dist-packages
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
Removing the /usr/lib/python2.7/dist-packages line causes /usr/lib/python2.7/dist-packages to be retrograded in sys.path and appear after ${HOME}/.local/lib/python2.7/site-packages.
I'm pretty sure I hadn't manually edited any .pth file. So I guess this may be the consequence of some python module I installed as root using easy-install. Which one?
From the modification times of the files and directories within /usr/local/lib/python2.7/dist-packages/, I can see that easy-install.pth has been modified the same day as two directories: astroid-1.3.2-py2.7.egg and pylint-1.4.0-py2.7.egg.
Next step will be to investigate why these modules' installation into /usr/local/lib/python2.7/dist-packages would require an early insertion of /usr/lib/python2.7/dist-packages into sys.path.
I'm confused about the way Python, on OS X, uses the packages in 'Extras' and 'site-packages'. In particular, I'm confused about the what I see in these directories, and how duplicate packages in the directories affect each other, and where I should put packages that I install.
I had assumed that 'Extras'1 was the location for packages that are not part of core Python, but that were nonetheless distributed with a given platform. For example, OS X, is distributed with PyObjC, twisted, and numpy, among others; and these are found in 'Extras'. I had also assumed that 'site-packages'2 is where packages that I subsequently install go, and that the directory is empty or absent on a "fresh" machine. Furthermore I'd assumed that installing a new package would leave the version in 'Extras' alone, and place any updates I do in 'site-packages', where they would "mask" those in 'Extras'. (So for example, the default installation could simply be restored by deleting 'site-packages'.)
What confuses me is that the contents of my 'Extras' directory is dwindling. Compared with the contents on a fresh OS X (10.8.2) install, the 'Extras' for my current configuration is missing
altgraph
altgraph-0.9-py2.7.egg-info
dateutil
macholib
macholib-1.4.2-py2.7.egg-info
modulegraph
modulegraph-0.9.1-py2.7.egg-info
numpy #though numpy-1.6.1-py2.7.egg-info is there
pkg_resources.py # though .pyc is there
py2app
py2app-0.6.3-py2.7.egg-info
setuptools # though setuptools-0.6c12dev_r88846-py2.7.egg-info is there
site.py # though .pyc and .pyo are there
xattr
xattr-0.6.2-py2.7.egg-info
zope
zope.interface-3.5.1-py2.7.egg-info
and has two additional files not found in the fresh install
pkg_resources.py.OLD.1356069438.31
setuptools-0.6c12dev_r88846-py2.7.egg-info.OLD.1356069438.31
setuptools.OLD.1356069438.31
To the best of my recollection, these are all packages that I've updated myself, and versions of all are present in 'site-packages'.
Is the contents of the 'Extras' directory supposed to "dwindle" in this way? Does updating a package that is in 'Extras' "move" it to 'site-packages', or should the 'Extras' versions be left alone and simply "masked" by those in 'site-packages'?
1:/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
2: /Library/Python/2.7/site-packages/
OSX (and Linux) does not seem to like to install extra python packages in the default installation directory of python distribution.
pip can search (like python itself does) for installed modules, and if you do an upgrade will de-install the old version and install the new version in the place you asked for, defaulting to site-packages. If you closely look at the output of pip it will tell you what it removed and what it installed.
So only if you tell pip to install to the Extras directory, or would have a pip patched to do so for OSX by default, that would work.
It is actually a good thing the older versions are removed, as otherwise the version of a module you import would be depending on your search path (sys.path) and that is kind of error prone.
Whether it is supposed to dwindle can be argued, but it is a normal result of the way you do your upgrades with pip.
Recently I found about this tool easy_install that help me to easy install additional python modules. The problem is that for each module it creates additional *.egg folder (sometime there is only an egg file?) (no source?) and I don't know how to setup eclipse paths.
By default I have included C:\Python26\Lib\site-packages and this is enough when I install python modules from source... but not when I'm using easy_intall
For example django instaled with easy_install is located in C:\Python26\Lib\site-packages\django-1.2.5-py2.6.egg\django and installed from source it's located in C:\Python26\Lib\site-packages\django
In fact when I'm using easy_install all installed modules are working without a problem, the only problem is that eclipse can't locate where is the source and gives me a false unresolved import errors
Where I'm wrong?
I'm assuming that eclipse does not search the egg files for source. Eggs, like jar files in Java, are just zipfiles of python code with some included metadata.
You'll also note that in site-packages you've got easy-install.pth and setuptools.pth files. Those files are parsed by python and used to add other directories and egg files to your PYTHONPATH (import sys; sys.path) so that Python can find the code in those locations. Eclipse isn't seeing those imports as valid because it is most likely not setup to take pth files into account.
To get Eclipse to recognize that Django is really installed you may want to try removing your easy_installed django package and reinstalling it with:
easy_install --always-unzip django
That way rather than installing a compressed egg file you'll have a normal package directory that eclipse should have a fairly easy time opening.
Alternatively, in your screenshot above it looks like you may just need to explicitly add each egg file you want eclipse to use.