I would like to create a library, say foolib, but to keep different subpackages separated, so to have barmodule, bazmodule, all under the same foolib main package. In other words, I want the client code to be able to do
import foolib.barmodule
import foolib.bazmodule
but to distribute barmodule and bazmodule as two independent entities. Replace module with package as well... ba[rz]module can be a fukll fledged library with complex content.
The reason behind this choice is manifold:
I would like a user to install only barmodule if he needs so.
I would like to keep the modules relatively independent and lightweight.
but I would like to keep them under a common namespace.
jQuery has a similar structure with the plugins.
Is it feasible in python with the standard setuptools and install procedure ?
You may be looking for namespace packages. See also PEP 382.
Yes, simply create a foolib directory, add an __init__.py to it, and make each sub-module a .py file.
/foolib
barmodule.py
bazmodule.py
then you can import them like so:
from foolib import barmodule
barmodule.some_function()
Related
I've made a small "library" (which is, in real, just 8 modules) to use in my projects.
How should I import part of them when create new project? Just copy "lib" folder with modules inside and print:
from lib import mod1
from lib import mod2
...
from lib import mod8
Is there a more "proper" way? I've tried zipping modules in one .pyz file using zipapp module in Python 3.5, but .pyz files can't be imported. I also can't upload modules to PyPi because only I'm going to use it (very specific task).
I've read something about Python eggs, but can't say for sure, whether that's a solution or not.
Maybe someone knows more "pythonic" way?
Depending on your needs, you can use pip to install your package from source control. That way you can host your library on your private git or svn or file share and list it in your package requirements the pythonic way instead of having to copy and paste it into each project that uses it.
As a side effect, you will also have an easy way to update it when you catch bugs and make enhancements.
I have a flask app that uses functions from custom modules.
My File hierarchy is like so:
__init__.py
ec2/__init__.py
citrixlb/__init__.py
So far in the root __init__.py I have a from ec2 import * clause to load my module.
Now I'm adding a new 'feature' called citrixlb.
Both the of the __init__.py files in citrixlb and ec2 use some of the same functions to do their task.
I was thinking of doing something like:
__init__.py
common/__init__.py
ec2/__init__.py
citrixlb/__init__.py
If I do the above,and move all common functions to common/__init__.py, how would ec2/__init__.py and citrixlb/__init__.py get access to the functions
in common/__init__.py?
The reason is that
I would like to keep the root __init__.py as sparse as possible
I wish to be able to run the __init__.py in citrixlb and ec2 as
standalone scripts.
I also wish to be able to continue to add functionality by adding newdir/__init__.py
If I do the above,and move all common functions to common/__init__.py, how would ec2/__init__.py and citrixlb/__init__.py get access to the functions in common/__init__.py?
This is exactly what explicit relative imports were designed for:
from .. import common
Or, if you insist on using import *:
from ..common import *
You can do this with absolute import instead. Assuming your top-level package is named mything:
from mything import common
from mything.common import *
But in this case, I think you're better with the relative version. It's not just more concise and easier to read, it's more robust (if you rename mything, or reorganize its structure, or embed this whole package inside a larger package…). But you may want to read the rationales for the two different features in PEP 328 to decide which one seems more compelling to you here.
One thing:
I wish to be able to run the __init__.py in citrixlb and ec2 as standalone scripts.
That, you can't do. Running modules inside a package as a top-level script is not supposed to work. Sometimes you get away with it. Once you're importing from a sibling or a parent, you definitely will not get away with it.
The right way to do it is either:
python -m mything.ec2 instead of python mything/ec2/__init__.py
Write a trivial ec2 script at the top level, that just does something like from mything.ec2 import main; main().
The latter is a common enough pattern that, if you're building a setuptools distribution, it can build the ec2 script for you automatically. And automatically make it still work even ec2 ends up in /usr/local/bin while the mything package is in your site-packages. See console_scripts for more details.
This question follows up The way to make namespace packages in Python and How do I create a namespace package in Python?.
Note PEP 420, and the distribute docs, which state:
You must NOT include any other code and data in a namespace package’s __init__.py. Even though it may appear to work during development, or when projects are installed as .egg files, it will not work when the projects are installed using “system” packaging tools – in such cases the __init__.py files will not be installed, let alone executed.
This all seems to make it impossible to have a "main library" package with independently distributed extension sub-packages. What I want is to be able to:
define a core library package, to be used like this:
import mylibrary
mylibrary.some_function()
allow library extensions, packaged and distributed separately, to be used like this:
import mylibrary.myextension
mylibrary.myextension.some_other_function()
I would've expected to be able to do this with namespace packages, but it seems not to be the case, based on the questions and links above. Can this be done at all?
It is indeed not possible to have code in a top level __init__.py for a PEP 420 namespace package.
If I were you, I'd either:
create 2 packages, one called mylibrary (a normal package) which contains your actual library code, and the other called mylibrary_plugins which is a namespace package.
or, create mylibrary.lib, which is a normal package and contains your code, and mylibrary.plugins, which is a namespace package.
Personally I'd use option 1.
The rationale section of PEP 420 explains why __init__.py cannot contain any code.
strictly speaking, you can have variables under mylibrary, you just won't be able to define them there. You can, for instance:
# mylibrary/core.py
import mylibrary
def some_function():
pass
mylibrary.some_function = some_function
and your users can use it like:
import mylibrary.core
mylibrary.some_function()
That is to say, mylibrary.core monkey patches mylibrary so that, other than the import, it looks as though somefunction is defined in mylibrary rather than a sub-package.
I currently have a project called "peewee" which consists of a single python file, peewee.py. There is also a module "tests.py" containing unit tests. This has been great, people that want to use the library can just grab a single file and run with it.
I've lately wanted to add some extras, but am not sure how to do this to make the namespacing right. If you look in the root of my project, it is something like:
peewee.py
tests.py
I want to add the following:
extras/__init__.py
extras/foo.py
extras/bar.py
And this is the tricky part. I want to have it such that folks using the single file can still do this, but if you want the extras you can have them, too. I want the extras to be namespaced such that:
from peewee.extras import foo
from peewee.extras.bar import Baz
My setup.py looks a bit like this:
setup(
name='peewee',
packages=['extras'],
py_modules=['peewee'],
# ... etc ...
)
But this doesn't quite work. Any help would be greatly appreciated!
Setting Up a Package
As #ThomasK said, the easiest way to do this would be with a package. If you name your package peewee, then you can edit the top-level __init__.py file to allow users to continue to use your package in the same way they have previously.
First, directory structure for your package and subfolders:
peewee/
__init__.py
peewee.py
extras/
__init__.py
foo.py
bar.py
The __init__.py file
Next, you need to add a few lines to the top-level __init__.py.
You could go for a quick-and-dirty method and just include:
from peewee.peewee import *
which would put everything in peewee.py in the top-level namespace of your package. Or, you could take the more traditional alternative and explicitly import only those methods that should be at the top level.
from peewee.peewee import funtion1, class1,...
and, for backwards compatibility, you could explicitly set the __all__ attribute of your module to include only peewee
__all__ = ['peewee']
which will let people continue to use from peewee import * if they really need to.
Writing a setup.py file
Finally, you'll have to set up some install scripts and such too. Zed Shaw's Learn Python The Hard Way exercise 46 has a simple and clear project skeleton that you should use.
The most important part is the setup.py file. The example page isn't too long and Zed's put a lot of work into making a really great book, so I'm not going to repost it here (though the entire book is available for free). You can also read the longer instructions/documentation for writing a setup.py file for distutils, however LPTHW will give you something that will do everything you want quickly and easily.
Total package directory structure
Note that your final directory structure will actually be a bit bigger (the name of peewee-pkg doesn't matter, bin is optional--the names of the subfolders matter)
peewee-pkg/
setup.py
bin
peewee/
__init__.py
peewee.py
extras/
__init__.py
foo.py
bar.py
Installing and using
After that, you could actually post your package to PyPi if you like, or you can distribute it to users directly. All they would need to do is run:
python setup.py install
and everything will be available to them.
Importing post-install
Finally, if you do specific imports in the peewee/__init__.py file as described earlier, your users will still be able to do:
from peewee import function1, class1, ...
But now they can also use import peewee.extras to get to the extras functions (or import peewee.extras.foo as foo or from pewee.extras.foo import extra_awesome), etc. And, as you asked in your question, they will also be able to do:
from pewee.extras import foo
And then access foo's functions as if the file were in the current directory.
Useful note for developing a package
On your computer, you should run:
python setup.py develop
which will install the package to your path just like using python setup.py install; however, develop tells python to recheck the file every time it uses the module, so that every time you make changes, they will be immediately available to the system for testing.
I've created python modules but they are in different directories.
/xml/xmlcreator.py
/tasklist/tasks.py
Here, tasks.py is trying to import xmlcreator but both are in different paths. One way to do it is include xmlcreator.py in the Pythonpath. But, considering that I'll be publishing the code, this doesn't seem the right way to go about it as suggested here. Thus, how do I include xmlcreator or rather any module that might be written by me which would be in various directories and sub directories?
Are you going to publish both modules separately or together in one package?
If the former, then you'll probably want to have your users install your xml module (I'd call it something else :) so that it is, by default, already on Python's path, and declare it as a dependency of the tasklist module.
If both are distributed as a bundle, then relative imports seem to be the best option, since you can control where the paths are relative to each other.
The best way is to create subpackages in a single top-level package that you define. You then ship these together in one package. If you are using setuptools/Distribute and you want to distribute them separately then you may also define a "namspace package" that the packages will be installed in. You don't need to use any ugly sys.path hacks.
Make a directory tree like this:
mypackage/__init__.py
mypackage/xml/__init__.py
mypackage/xml/xmlcreator.py
mypackage/tasklist/__init__.py
mypackage/tasklist/tasks.py
The __init__.py files may be empty. They define the directory to be a package that Python will search in.
Except if you want to use namespace packages the mypackage/__init__.py should contains:
__import__('pkg_resources').declare_namespace(__name__)
And your setup.py file contain:
...
namespace_packages=["mypackage"],
...
Then in your code:
from mypackage.xml import xmlcreator
from mypackage.tasklist import tasks
Will get them anywhere you need them. You only need to make one name globally unique in this case, the mypackage name.
For developing the code you can put the package in "develop mode", by doing
python setup.py develop --user
This will set up the local python environment to look for your package in your workspace.
When I start a new Python project, I immediately write its setup.py and declare my Python modules/packages, so that then I just do:
python setup.py develop
and everything gets magically added to my PYTHONPATH. If you do it from a virtualenv it's even better, since you don't need to install it system-wide.
Here's more about it:
http://packages.python.org/distribute/setuptools.html#development-mode