This should be a common scenario, but could not find any relevant post yet..
I plan to deploy a Python library (I guess the same applies to regular applications) which makes use of some images and other resource files. What is the standard location for such items? I imagine, for project Foo, the choices would be
Have resources directory in the source repository and then move files to /usr/share/foo/
Place resources directly inside the python package that goes under /usr/lib/python-<version>/foo/
Any suggestions?
Edit: As suggested, clarifying that the main platform this will be running on is Linux.
This question is somewhat incomplete, because a proper answer would depend on the underlying operating system, as each has its own modus operandi. In linux (and most unix based OSs) for example /usr/share/foo or /usr/local/share/foo would be the standard. In OS X you can do the same, but I would think "/Library/Application Support/Foo" (although that's usually for storing settings and whatnot) would be the place to put such things, though if you're writing libraries following the "Framework" idea, all the resources would be included in the /Library/Frameworks/Foo.Framework" ... Apps on OS X on the other hand should keeps all there resources within the Resources directory inside Foo.app
We put non .py files in /opt/foo/foo-1.2/...
Except, of course, for static media that is served by Apache, that goes to /var/www/html/foo/foo-1.1/media/...
Except, of course, for customer-specific configuration files. They go to
/var/opt/customer/foo/...
Those follow the Linux standards as I understand them.
We try to stay away from /usr/lib/ and /lib kinds of locations because those feel like they're part of the distribution. We lean toward /opt and /var because they're clearly separated from the linux distro directories.
The standard location is where your standard libs goes. But it doesn't sound to me from what you've written, that you'll want your python lib there. I think you should try out Virtualenv.
If you don't want to go through all the trouble (well, it really just amounts to sudo easy_install virtualenv for you), you could try to just dump your python lib in any dir in your ~/ and do something along the lines of
import sys
sys.path.append( '/full/path/to/your/lib/goes/here')
to any given application that uses your lib.
Please bear in mind, that the examples given are for test-purposes only. For anything live-ish, I would recommend that you use distutil. Examples of use are given here.
Related
I want to know if I can create a python script with a folder in the same directory with all the assets of a python module, so when someone wants to use it, they would not have to pip install module, because it would import from the directory.
Yes, you can, but it doesn't mean that you should.
First, ask yourself who is suposed to use that code.
If you plan to give it to consumers, it would be a good idea to use a tool like py2exe and create executable file which would include all modules and not allow for code to be changed.
If you plan to share it with another developer, you might want to look into virtual environments and requirements.txt file.
There are multiple reasons why sharing modules is bad idea:
It is harder to update modules later, at least without upgrading whole project.
It uses more space on version control, which can create issues on huge projects with hundreds of modules and branches
It might be illegal as some licenses specifically forbid including their code in your source code.
The pip install of some module might do different things depending on operating system version or installed packages. The modules on your machine might be suboptimal on someone else's machine, and in some instances might not even work.
And probably more that I can't think of right now.
The only situation where I saw this being unavoidable was when the module didn't support python implementation the application was running on. The module was changed, and its source was put under lib folder with the rest of the libraries.
I think you can add the directory with python modules into PYTHONPATH. Then people want to use those modules just need has this envvar set.
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH
I would like to gather many libraries I have made while working on my projects in some kind of container, so that I can easily use any of them in future projects of mine. It is pretty clear to me how to do this, except one part.
I am assuming that every service will have its own config file (for instance, the Cache service, will have a config file with cache host and port, and so on). Now the problem is: when I want to use this container in an arbitrary project I will have to make assumptions about the project directory structure to know where to find these config files.
For instance, one might assume that on the same path of my library there is a config folder where I will find the config files of my services. However, this might conflict with the project's directory structure (i.e. the project might already have its own config directory for instance).
So, all in all, my question is: is there a safe, standard way to ship a library which might assume to find some config files someplace, or for which example config files are shipped along with the library itself?
well, you should not keep config files, or anything that you want to modify along with code in python (or actually in any language). Each OS have folders for that purpose.
Either it's system wide, and on Unix it's /etc or it's for an user it's in ~/.config. You have theLibrary folders for OSX, and I'm sure there's something alike for windows beyond \Windows\SYSTEM32 😉.
What that means is that the path to your configuration files shall not be considered relative to your code at any point. Never. Ever.
You can include some assets in a python package, using the MANIFEST.in but, as it'll be within your python package, you shall assume you won't have rights to write where it'll be (installed by admin, ran by user).
You can also specify some of those assets to install at specific places using setup.py's data_files directive, which will be installed relatively to sys.prefix.
Common practice is to provide configuration files examples using a link from the documentation, or better generate those files when starting the application.
Also, another trend for desktops, is to use the XDG directory specification, to decide where to look for, or where to place your configuration files.
To sum it up:
make a list of default paths your code expects to find the configuration,
make it possible to specify manually at command line the path to the configuration python foo.py --config bar.ini
write a feature for your tool to generate the configuration (with a series of questions)
deploy your default configurations to standard places (XDG paths, $prefix/etc…)
I'm a self-taught, amateur, purely recreational programmer. I don't understand all the fancy programming lingo, and I certainly don't have any good resources, apart from this website, where I can go for help. (i.e., Please dumb it down for me!) I would imagine my question here is somewhat common, but I honestly couldn't find any answers on Google or this website, probably because I don't know the proper terminology to search for.
~~~
Having said that, I feel I have a pretty solid grasp on the basics of Python. And now, I've created an application that I'd like to share with a friend. My application accesses JPEG image files on my computer using a directory path that I've written into the code itself. However, I'd like my friend to be able to store these image files anywhere on their computer, not necessarily in the file folder that I've been using.
I assume the best way to accomplish this is to allow my friend to choose the directory path for themselves and then to write their chosen directory path to a file at a predetermined location on their computer. My application would then have that file's location prewritten into its code. This way, it would be trivially easy to open the file at the predetermined location, and then that file would point my application to my friend's chosen directory path.
1.) Are any of my intuitions here misguided? Are there better ways of doing this?
2.) If you think my general approach is a reasonable one, then is there a good/common place on the computer where applications typically store their directory paths upon installation?
Any advice - or any recommended resources - would be very much appreciated! Thanks!
Well, the standard way to do this is a lot more complicated and platform-specific:
On traditional Unix, this is pretty simple; you create a text file in some simpler format (e.g., that used by ConfigParser, named, say, ~/.myprogram.cfg, and you write a line to it that looks like image_path=/path/to/images.
On most modern Linux systems, or any other FreeDesktop/XDG-based system, you should (at least for GUI apps) instead use a special directory looked up in the environment as XDG_CONFIG_HOME, falling back to ~/.config, instead of using ~.
On Windows, the standard place to store stuff like this is the Windows Registry (e.g., by using winreg), by creating a key for your program and storing a value with name image_path and value /path/to/images there.
On Mac, the standard place to store stuff like this is in the NSUserDefaults database (e.g., by using PyObjC, which isn't part of the stdlib but does come built-in with Apple's pre-installed Python) by opening the default domain for your program and adding a value with key image_path and value… well, you probably want a Cocoa bookmark (maybe even a security-scoped one), not a path.
That probably all sounds way, way too complicated.
One option is to use a library that wraps this all up for you. If you're already using a heavy-duty framework like, say, Qt, it probably has functionality built-in to do that. Otherwise, it may take a lot of searching to find something.
A simpler alternative is to just pretend everything is like traditional Unix. That will work on Windows and Mac. It will be slightly annoying on some Windows versions that your config file will be visible in their home directory, but not a huge deal. It means you won't get some of the bonus features that Mac provides, like being able to magically follow the directory if the user moves it somewhere else on his hard drive, or remembering the settings if he reinstalls OS X and migrates his old settings, but again, usually that's fine.
In between the extremes, you can pretend everything is like Linux, using a special, and unobtrusive, location for the files on Windows and Mac just as you do there. Both platforms have APIs to look up special directories, called "application data" on Windows and "application support" on Mac. Using PyWin32 or PyObjC, respectively, these are pretty easy to look up. (For example, see this answer.) Then you just create a subdirectory there named My App on Windows, or com.mydomain.myapp on Mac, and store the file there.
I need to ship a collection of Python programs that use multiple packages stored in a local Library directory: the goal is to avoid having users install packages before using my programs (the packages are shipped in the Library directory). What is the best way of importing the packages contained in Library?
I tried three methods, but none of them appears perfect: is there a simpler and robust method? or is one of these methods the best one can do?
In the first method, the Library folder is simply added to the library path:
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'Library'))
import package_from_Library
The Library folder is put at the beginning so that the packages shipped with my programs have priority over the same modules installed by the user (this way I am sure that they have the correct version to work with my programs). This method also works when the Library folder is not in the current directory, which is good. However, this approach has drawbacks. Each and every one of my programs adds a copy of the same path to sys.path, which is a waste. In addition, all programs must contain the same three path-modifying lines, which goes against the Don't Repeat Yourself principle.
An improvement over the above problems consists in trying to add the Library path only once, by doing it in an imported module:
# In module add_Library_path:
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'Library'))
and then to use, in each of my programs:
import add_Library_path
import package_from_Library
This way, thanks to the caching mechanism of CPython, the module add_Library_path is only run once, and the Library path is added only once to sys.path. However, a drawback of this approach is that import add_Library_path has an invisible side effect, and that the order of the imports matters: this makes the code less legible, and more fragile. Also, this forces my distribution of programs to inlude an add_Library_path.py program that users will not use.
Python modules from Library can also be imported by making it a package (empty __init__.py file stored inside), which allows one to do:
from Library import module_from_Library
However, this breaks for packages in Library, as they might do something like from xlutils.filter import …, which breaks because xlutils is not found in sys.path. So, this method works, but only when including modules in Library, not packages.
All these methods have some drawback.
Is there a better way of shipping programs with a collection of packages (that they use) stored in a local Library directory? or is one of the methods above (method 1?) the best one can do?
PS: In my case, all the packages from Library are pure Python packages, but a more general solution that works for any operating system is best.
PPS: The goal is that the user be able to use my programs without having to install anything (beyond copying the directory I ship them regularly), like in the examples above.
PPPS: More precisely, the goal is to have the flexibility of easily updating both my collection of programs and their associated third-party packages from Library by having my users do a simple copy of a directory containing my programs and the Library folder of "hidden" third-party packages. (I do frequent updates, so I prefer not forcing the users to update their Python distribution too.)
Messing around with sys.path() leads to pain... The modern package template and Distribute contain a vast array of information and were in part set up to solve your problem.
What I would do is to set up setup.py to install all your packages to a specific site-packages location or if you could do it to the system's site-packages. In the former case, the local site-packages would then be added to the PYTHONPATH of the system/user. In the latter case, nothing needs to changes
You could use the batch file to set the python path as well. Or change the python executable to point to a shell script that contains a modified PYTHONPATH and then executes the python interpreter. The latter of course, means that you have to have access to the user's machine, which you do not. However, if your users only run scripts and do not import your own libraries, you could use your own wrapper for scripts:
#!/path/to/my/python
And the /path/to/my/python script would be something like:
#!/bin/sh
PYTHONPATH=/whatever/lib/path:$PYTHONPATH /usr/bin/python $*
I think you should have a look at path import hooks which allow to modify the behaviour of python when searching for modules.
For example you could try to do something like kde's scriptengine does for python plugins[1].
It adds a special token to sys.path(like "<plasmaXXXXXX>" with XXXXXX being a random number just to avoid name collisions) and then when python try to import modules and can't find them in the other paths, it will call your importer which can deal with it.
A simpler alternative is to have a main script used as launcher which simply adds the path to sys.path and execute the target file(so that you can safely avoid putting the sys.path.append(...) line on every file).
Yet an other alternative, that works on python2.6+, would be to install the library under the per-user site-packages directory.
[1] You can find the source code under /usr/share/kde4/apps/plasma_scriptengine_python in a linux installation with kde.
I'm developing python C++ extensions for use in both OSX and linux. Currently, I can run my code with a wrapper script wrapper.sh:
#!/bin/bash
trunk=`dirname $0`
trunk=`cd $trunk; pwd`
export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:$trunk/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$trunk/lib/:$trunk/src/hdf5/lib/:$trunk/src/python/lib
$trunk/src/python/bin/python "$#"
which is able to set up my run like this: wrapper.sh app.py
What I would like to do is to eliminate the need for wrapper.sh, so I need alternatives for DYLD_LIBRARY_PATH and LD_LIBRARY_PATH. I can not put my libraries in some standard location like /usr/local/lib because on my machine, I maintain several independent instances of my libraries. That is, my libraries need to be kept somewhere relative to my installation path. I can't put these environment variables in my login script for the same reason. Currently, I need to call one of my wrapper.sh scripts to use the associated libraries. My goal is to be able to run merely app.py, which if it lives in my installation path, should be able to find its associated python and libraries. The purpose is to simplify execution for users, and to simplify usage of external tools like nosetests.
One alternative seems to be using rpath when I build my version of python:
./configure --enable-shared --prefix=$(CURDIR)/$(PYTHON_DIR) LDFLAGS="-Wl,-rpath,$(CURDIR)/lib/ -Wl,-rpath,$(CURDIR)/src/hdf5/lib -Wl,-rpath,$(CURDIR)/src/python/lib"
This trick seems to work fine on linux, even though one of my libraries ended up needing to be copied directly into trunk/src/python/lib/python2.6/lib-dynload for some reason unclear to me. However, this trick is not working on OSX; it looks like I need to run install_name_tool on all my dylibs libraries.
The other alternative I came up with was to do something like this:
ln -s wrapper.sh python
so that my scripts could all use #! ../python, but I'm getting Unmatched ". errors. Same thing if I use #! ../wrapper.sh. I'm not really an expert in bash...
However, these all seem so unnecessarily complicated, and surely this is something that other people have solved?? Thanks for any advice!
For python extensions, consider using PYTHONPATH: the Python interpreter will search the PYTHONPATH for .py/.pyc/.pyo/.so modules, as well as packages. See docs for Python 2.x as well as docs for Python 3.x; specifically the section named "The Module Search Path" on both pages. This also references information that seems to indicate that it is possible to update the module search path at runtime, which, if true, means that you could add all that logic to your program and it can hunt for its libraries on its own (say if it installs a copy in /usr/libexec/pkgname/... somewhere or something).
For all but the most complex of cases, though, setting PYTHONPATH and using a shell-script or native-compiled binary wrapper to start the core program is an okay approach, and one that is also used in other language environments including Mono and Java.
Not sure if this would be an acceptable (partial) solution in your circumstances, but another way to get libraries noticed by ld on linux is to add the path to the libraries to /etc/ld.so.conf and then runldconfig
For the Mac I don't remember the details, but I think Apple provide some resources for distributing apps packaged as a .app which includes some default locations (relative to the root of the .app) for libraries, or "frameworks" as they call them. Would require some googling from there - sorry can't help further on that but hope you get some progress :-)