In a lot of Python libraries I see a module called "ext", for example sqlalchemy.ext. I was just curious what the abbreviation means and what the module is usually used for.
Generally it is used to namespace external packages that are not part of the core but provide added functionality.
I like the rational that Flask uses. They use ext as a generic namespace proxy to load external modules.
On import flask.ext.foo, it first tries to find flask_foo, then flaskext.foo
This makes it easier for end users to remember a naming pattern and lets the library figure out where to load the extension from.
Extended or extension, i.e. features / functionalities beyond the core.
Related
Below is a screenshot of part of an article explaining how to access the example Python module dataset.py, for which they provide the following line:
import my_model.training.dataset
I'd like to know if the following methods below are equivalent and accomplish the same thing:
from my_model.training import dataset
from my_model import training.dataset
I have a library where I've been accumulating all of my .py files over time. I'm trying to organize it into something more.. neat but I'm having trouble deciding how to do that.
The library (or rather, the folder I'm dumping everything in) is meant to be just a collection of independent modules, but some of the modules have cross dependencies.. It'd be easier if I had a systematic way to group functions/classes within certain files ie modules. Should they be grouped by purpose?
keep in mind these aren't even packages for projects, they are the building blocks for other packages; just my own personal collection of classes and functions but starting to get hard to manage. so i could use some advice
Thanks
Is there a way to perform keyword searching of module and function docstrings from the interpreter?
Often, when I want to do something in Python, I know there's a module that does what I want, but I don't know what it's called. I would like a way of searching for "the name of the function or module that does X" without having to Google "python do X".
Take the example of "how can I open an URL"? At a Linux shell, I might try >> apropos open url. Under MATLAB, I might try >> lookup open url. Both of these would give me listings of functions or modules that include the words 'open' and 'URL' somewhere in their man page or doc string. For example:
urllib.urlopen : Create a file-like object for the specified URL to read from.
urllib2.urlopen : ...
...
I'd like something that searches through all installed modules, not just the modules that have been imported into my current session.
Yes, Google is a great way to search Python doc strings, but the latency is a bit high. ;)
The built-in support for that comes from pydoc.apropos:
import pydoc
pydoc.apropos('Zip')
# output: zipimport - zipimport provides support for importing Python modules from Zip archives.
Which, as you can see, is nearly useless. It also stops working whenever a module cannot be imported, which might mean 'always' depending on your package management style.
An alternative that I haven't used but looks promising is apropos.py:
Deep, dirty, and exhaustive 'apropos' for Python. Crawls all
libraries in the system.path, and matches a querystring against all
module names, as well as class, function, and method names and
docstrings of the top-level modules.
Usage: ./apropos.py
This module was created due to the limits of PyDoc's apropos method,
which fails hard when one of the top level modules cannot be imported.
The native apropos method also does not crawl docstrings, or deep
module names, which this module does.
Use the following command:
pydoc -k
I have this core python module we use in our facility called mfxLib. I need to be able to keep different version of this module without breaking all the other modules/plugin that are importing this module.
My solution was keep a duplicate of my module by renaming them mfxLib01 and mfxLib02 then
to replace the original mfxLib module with an empty module containing only a __init__.py file that import the latest version.
# content of mfxLib.__init__.py
from mfxLib02 import *
This seems logical and seems to work but I was wondering if there was a common practice for doing this? guidelines to follow? etc
Thanks
You can import a module as another name. Commonly people use this to save typing in a long module name, for example:
import numpy as np
np.array([1,2,3,4])
Hence you could do:
import mfxLib01 as mfxLib
or
import mfxLib02 as mfxLib
then your code uses mfxLib everywhere.
That might help...
If you have different scripts requiring different versions, your current approach should be the the best, but I'd suggest using a version control system like Git or SVN. That would allow you to commit and revert to earlier versions easily, as well as share the module with other users.
Version control will almost certainly make your life easier. In addition to Petterson's recommendations, consider Mercurial. Like git and SVN, it's free. It's also written in Python and should run without difficulty on any of your systems.
Spacedman's recommendations are also useful, especially if the differences between the versions represent customizations for particular systems and the customizations are relatively stable. Note that you can use that approach in combination with a version control system.
Finally, it's always worthwhile to make a strong effort to write your module so that it can work without modification everywhere. Often, you can accomplish this by adding some optional arguments to a few key functions to handle the different requirements. Python is really convenient in that regard because keyword arguments at the end of the arg list are always optional, so you can easily arrange to provide the existing behavior by giving them suitable default values.
def foo(oldarg1, oldarg2, newarg1=None):
if newarg1 != None:
## behave differently
else:
## behave as usual
Ruby uses require, Python uses import. They're substantially different models, and while I'm more used to the require model, I can see a few places where I think I like import more. I'm curious what things people find particularly easy — or more interestingly, harder than they should be — with each of these models.
In particular, if you were writing a new programming language, how would you design a code-loading mechanism? Which "pros" and "cons" would weigh most heavily on your design choice?
The Python import has a major feature in that it ties two things together -- how to find the import and under what namespace to include it.
This creates very explicit code:
import xml.sax
This specifies where to find the code we want to use, by the rules of the Python search path.
At the same time, all objects that we want to access live under this exact namespace, for example xml.sax.ContentHandler.
I regard this as an advantage to Ruby's require. require 'xml' might in fact make objects inside the namespace XML or any other namespace available in the module, without this being directly evident from the require line.
If xml.sax.ContentHandler is too long, you may specify a different name when importing:
import xml.sax as X
And it is now avalable under X.ContentHandler.
This way Python requires you to explicitly build the namespace of each module. Python namespaces are thus very "physical", and I'll explain what I mean:
By default, only names directly defined in the module are available in its namespace: functions, classes and so.
To add to a module's namespace, you explicitly import the names you wish to add, placing them (by reference) "physically" in the current module.
For example, if we have the small Python package "process" with internal submodules machine and interface, and we wish to present this as one convenient namespace directly under the package name, this is and example of what we could write in the "package definition" file process/__init__.py:
from process.interface import *
from process.machine import Machine, HelperMachine
Thus we lift up what would normally be accessible as process.machine.Machine up to process.Machine. And we add all names from process.interface to process namespace, in a very explicit fashion.
The advantages of Python's import that I wrote about were simply two:
Clear what you include when using import
Explicit how you modify your own module's namespace (for the program or for others to import)
A nice property of require is that it is actually a method defined in Kernel. Thus you can override it and implement your own packaging system for Ruby, which is what e.g. Rubygems does!
PS: I am not selling monkey patching here, but the fact that Ruby's package system can be rewritten by the user (even to work like python's system). When you write a new programming language, you cannot get everything right. Thus if your import mechanism is fully extensible (into totally all directions) from within the language, you do your future users the best service. A language that is not fully extensible from within itself is an evolutionary dead-end. I'd say this is one of the things Matz got right with Ruby.
Python's import provides a very explicit kind of namespace: the namespace is the path, you don't have to look into files to know what namespace they do their definitions in, and your file is not cluttered with namespace definitions. This makes the namespace scheme of an application simple and fast to understand (just look at the source tree), and avoids simple mistakes like mistyping a namespace declaration.
A nice side effect is every file has its own private namespace, so you don't have to worry about conflicts when naming things.
Sometimes namespaces can get annoying too, having things like some.module.far.far.away.TheClass() everywhere can quickly make your code very long and boring to type. In these cases you can import ... from ... and inject bits of another namespace in the current one. If the injection causes a conflict with the module you are importing in, you can simply rename the thing you imported: from some.other.module import Bar as BarFromOtherModule.
Python is still vulnerable to problems like circular imports, but it's the application design more than the language that has to be blamed in these cases.
So python took C++ namespace and #include and largely extended on it. On the other hand I don't see in which way ruby's module and require add anything new to these, and you have the exact same horrible problems like global namespace cluttering.
Disclaimer, I am by no means a Python expert.
The biggest advantage I see to require over import is simply that you don't have to worry about understanding the mapping between namespaces and file paths. It's obvious: it's just a standard file path.
I really like the emphasis on namespacing that import has, but can't help but wonder if this particular approach isn't too inflexible. As far as I can tell, the only means of controlling a module's naming in Python is by altering the filename of the module being imported or using an as rename. Additionally, with explicit namespacing, you have a means by which you can refer to something by its fully-qualified identifier, but with implicit namespacing, you have no means to do this inside the module itself, and that can lead to potential ambiguities that are difficult to resolve without renaming.
i.e., in foo.py:
class Bar:
def myself(self):
return foo.Bar
This fails with:
Traceback (most recent call last):
File "", line 1, in ?
File "foo.py", line 3, in myself
return foo.Bar
NameError: global name 'foo' is not defined
Both implementations use a list of locations to search from, which strikes me as a critically important component, regardless of the model you choose.
What if a code-loading mechanism like require was used, but the language simply didn't have a global namespace? i.e., everything, everywhere must be namespaced, but the developer has full control over which namespace the class is defined in, and that namespace declaration occurs explicitly in the code rather than via the filename. Alternatively, defining something in the global namespace generates a warning. Is that a best-of-both-worlds approach, or is there an obvious downside to it that I'm missing?
I have a module whose purpose is to define a class called "nib". (and a few related classes too.) How should I call the module itself? "nib"? "nibmodule"? Anything else?
Just nib. Name the class Nib, with a capital N. For more on naming conventions and other style advice, see PEP 8, the Python style guide.
I would call it nib.py. And I would also name the class Nib.
In a larger python project I'm working on, we have lots of modules defining basically one important class. Classes are named beginning with a capital letter. The modules are named like the class in lowercase. This leads to imports like the following:
from nib import Nib
from foo import Foo
from spam.eggs import Eggs, FriedEggs
It's a bit like emulating the Java way. One class per file. But with the added flexibility, that you can allways add another class to a single file if it makes sense.
I know my solution is not very popular from the pythonic point of view, but I prefer to use the Java approach of one module->one class, with the module named as the class.
I do understand the reason behind the python style, but I am not too fond of having a very large file containing a lot of classes. I find it difficult to browse, despite folding.
Another reason is version control: having a large file means that your commits tend to concentrate on that file. This can potentially lead to a higher quantity of conflicts to be resolved. You also loose the additional log information that your commit modifies specific files (therefore involving specific classes). Instead you see a modification to the module file, with only the commit comment to understand what modification has been done.
Summing up, if you prefer the python philosophy, go for the suggestions of the other posts. If you instead prefer the java-like philosophy, create a Nib.py containing class Nib.
nib is fine. If in doubt, refer to the Python style guide.
From PEP 8:
Package and Module Names
Modules should have short, all-lowercase names. Underscores can be used
in the module name if it improves readability. Python packages should
also have short, all-lowercase names, although the use of underscores is
discouraged.
Since module names are mapped to file names, and some file systems are
case insensitive and truncate long names, it is important that module
names be chosen to be fairly short -- this won't be a problem on Unix,
but it may be a problem when the code is transported to older Mac or
Windows versions, or DOS.
When an extension module written in C or C++ has an accompanying Python
module that provides a higher level (e.g. more object oriented)
interface, the C/C++ module has a leading underscore (e.g. _socket).
From PEP-8: Package and Module Names:
Modules should have short, all-lowercase names. Underscores can be
used in the module name if it improves readability.
Python packages should also have short, all-lowercase names, although the use of
underscores is discouraged.
When an extension module written in C or C++ has an accompanying
Python module that provides a higher level (e.g. more object oriented)
interface, the C/C++ module has a leading underscore (e.g. _socket).