I implemented a python extension module in C according to https://docs.python.org/3.3/extending/extending.html
Now I want to have integer constants in that module, so I did:
module= PyModule_Create(&myModuleDef);
...
PyModule_AddIntConstant(module, "VAR1",1);
PyModule_AddIntConstant(module, "VAR2",2);
...
return module;
This works. But I can modify the "constants" from python, like
import myModule
myModule.VAR1 = 10
I tried to overload __setattr__, but this function is not called upon assignment.
Is there a solution?
You can't define module level "constants" in Python as you would in C(++). The Python way is to expect everyone to behave like responsible adults. If something is in all caps with underscores (like PEP 8 dictates), you shouldn't change it.
Related
I have two related Python 'import' questions. They are easily testable, but I want answers that are language-defined and not implementation-specific, and I'm also interested in style/convention, so I'm asking here instead.
1)
If module A imports module B, and module B imports module C, can code in module A reference module C without an explicit import? If so, am I correct in assuming this is bad practice?
2)
If I import module A.B.C, does that import modules A and A.B as well? If so, is it by convention better to explicitly import A; import A.B; import A.B.C?
The first thing you should know is that the Python language is NOT an ISO standard. This is rather different from C/C++, and it means that there's no "proper" way to define a language behaviour - CPython might do something just because it was coded that way, and Jython might do the other way round.
about your questions, remember that "importing" a module is a two-part operation: first the module is loaded - if it had never been, e.g. if it wasn't available in sys.modules, then a name is bound to that module in the local namespace.
hence:
1) Yes, you can reference whatever you want from module a by providing the proper namespace, e.g. you'll have to do something like
B.C.name = "something"
And I think this is very rarely done in Python programs and could be considered bad practice since it forces a "transitive dep" - if some module B implementation is refactored and doesn't depend on C anymore, it should continue offering the C module just for satisfying A deps.
Of course setting __ all __ can prevent this, and a good practice may be to put __ all __ in all your modules, and export just the symbols you want to be really public.
2) Yes and no. Doing
import a.b.c.d
performs the first import phase (loading) on all modules, but the second just on a (and, recursively, in b with respect to c, etc) but all the modules in the chain must be referenced by full namespace; after such an import, you can do
a.something
a.b.something
a.b.c.something
but you can't do
c.something
b.something
I must admit that kind of usage is pretty rare as well; I generally prefer the "from module import something" way-to-import, and generally you just ask for what you need - such nesting is neither common in libraries, nor its usage is that common.
Many times there're "outer packages", just used for organization, which hold modules with classes. It's very likely that a, b, c above are just packages, and d is a module which truly holds classes, functions and other objects. So the proper usage would be:
from a.b.c.d import name1, name2, name3
I hope this satifies your curiosity.
Alan's given a great answer, but I wanted to add that for your question 1 it depends on what you mean by 'imports'.
If you use the from C import x syntax, then x becomes available in the namespace of B. If in A you then do import B, you will have access to x from A as B.x.
It's not so much bad practice as potentially confusing, and will make debugging etc harder as you won't necessarily know where the objects have come from.
I would like to assign an another function to len in __init__.py file of my package the following way:
llen = len
len = lambda x: llen(x) - 1
It works fine, but only in the __init__.py file. How can I make it affect other modules in my package?
This may not be the answer you're looking for, but I wouldn't do this if I were you (and I'm pretty sure you can't easily, anyway).
The reason why you shouldn't is that python uses len internally on it's objects to perform certain operations.
Another reason is pure broken logic. Your len function defined above would return a negative length for empty lists, or empty things. This seems quite broken to me.
What you can do, is override the length method only on certain classes (this might make a lot of sense for you). For this you can use operator overloading, just override the method __len__ in your class:
class MyList(object):
def __len__(self,):
# Do your thing
You may also want to look into meta classes, there is a very good stack overflow question on this subject.
When you try to load a name that is not defined as a module-level global or a function local, Python looks it up in the __builtin__(builtins in Python 3) module. In both versions of Python, this module is also availabe as __builtins__ in the global scope. You can modify this module and this will affect not only your code but any python code anywhere that runs after your code runs!!
import __builtin__ as builtins # import builtins in python 3
llen = len
builtins.len = lambda a:llen(a) - 1
I see that there are other questions related to cross module variables, but they don't really fully answer my question.
I have an application that I have split into 3 modules + 1 main application, mainly for ease of readability and maintainability.
2 of these modules have threads with variables that need to be modified from other modules and other module threads.
Whilst I can modify a module's variable from the main code, I don't appear to be able to modify one module's variable from another module unless I import every module into every other module.
The example below where a&b are imported into main and a module a needs to access a variable in module b:
main
module a
var a
module b
var a
main
a.a = 1
b.a = 2
module a
b.a = 3
module b
a.a = 0
without importing module a into module b and importing module b into module a, can this be achieved globally through the main program ?
If I do have to import a and b into main, and then import a into b and b into a, what are the implications in terms of memory and resource usage / speed etc ?
I tried the suggestion from #abarnert:
#moda
vara = 10
#modb
print(str(vara))
#main
import moda
from moda import vara
import modb
however I get "name error vara is not defined"
If the code in the modules are defined as classes, and the main program creates instances of these classes, the main program can pass an instance of one module class to another, and changes to that instance will be reflected everywhere. There would be no need to import a or b into each other, because they would simply have references to each other.
If I do have to import a and b into main, and then import a into b and b into a, what are the implications in terms of memory and resource usage / speed etc ?
Absolutely none for memory—every module that imports a will get a reference to the exact same a module object. All you're doing is increasing its refcount, not creating new objects.
For speed, the time to discover that you're trying to import a module that already exists is almost nothing (it's just looking up the module name in a dictionary). It is slightly slower to access a.a than to just access a. But this is very rarely an issue. If it is, you're almost certainly going to want to copy that value into the locals of whatever function is accessing it over and over, at which point it won't matter which globals it came from.
without importing module a into module b and importing module b into module a, can this be achieved globally through the main program ?
Sure. All you have to do is import (with from a import a or import a.a as aa or whatever) or copy the variables from module a into main.
Note that just makes a new name for each value; it doesn't make references to the variables. There is no such thing as a reference to a variable in Python.
This works if the variables are holding constants, or if they're holding mutable values that you modify. It just doesn't do anything useful if the variables are names that you want to rebind to new values. (If you do need to do that, just wrap the values in something mutable—e.g., turn each variable into a 1-item list, so you can rebind a[0] instead of a, which means anyone else who has a reference to a can see your new a[0] value.)
If you insist on a "true global", even that isn't impossible. See builtins for details. But you almost certainly don't want this.
If you want to be able to modify a module-level variable from a different module then yes, you will need to import the other module. I would question why you need to do this. Perhaps you should be breaking your code into classes instead of separate modules.
For example you could choose to encapsulate the variables that need to be modified by both modules inside a separate class and pass a single instance of that class to all classes (or modules but you should really use classes) that need it.
See Circular (or cyclic) imports in Python for more information about cyclical imports.
I've been looking through the code of the os module (just to be clear, I'm looking at the file /usr/lib/python2.7/os.py), and I've been trying to find the code for the mkdir function. From what I could tell, it comes from the 'posix' module, and its a built-in function, same as range or max:
>>> import posix
>>> posix.mkdir
<built-in function mkdir>
>>> max
<built-in function max>
I'm guessing the code for these is written in C somewhere, and the python interpreter knows where to find them. Could someone explain, or point me to some resources that do, how and where these built-in function are written and how they are integrated with the interpreter?
Thanks!
On POSIX platforms (and on Windows and OS/2) the os module imports from a C module, defined in posixmodule.c.
This module defines a posix_mkdir() function that wraps the mkdir() C call on POSIX platforms, CreateDirectoryW on Windows.
The module registers this function, together with others, in the module PyMethodDef posix_methods structure. When the module is imported, Python calls the PyMODINIT_FUNC() function, which uses that structure to create an approriate module object with the posix_methods structure and adds a series of constants (such as the open() flag constants) to the module.
See the Extending Python with C or C++ tutorial on how C extensions work.
In python, if you need a module from a different package you have to import it. Coming from a Java background, that makes sense.
import foo.bar
What doesn't make sense though, is why do I need to use the full name whenever I want to use bar? If I wanted to use the full name, why do I need to import? Doesn't using the full name immediately describe which module I'm addressing?
It just seems a little redundant to have from foo import bar when that's what import foo.bar should be doing. Also a little vague why I had to import when I was going to use the full name.
The thing is, even though Python's import statement is designed to look similar to Java's, they do completely different things under the hood. As you know, in Java an import statement is really little more than a hint to the compiler. It basically sets up an alias for a fully qualified class name. For example, when you write
import java.util.Set;
it tells the compiler that throughout that file, when you write Set, you mean java.util.Set. And if you write s.add(o) where s is an object of type Set, the compiler (or rather, linker) goes out and finds the add method in Set.class and puts in a reference to it.
But in Python,
import util.set
(that is a made-up module, by the way) does something completely different. See, in Python, packages and modules are not just names, they're actual objects, and when you write util.set in your code, that instructs Python to access an object named util and look for an attribute on it named set. The job of Python's import statement is to create that object and attribute. The way it works is that the interpreter looks for a file named util/__init__.py, uses the code in it to define properties of an object, and binds that object to the name util. Similarly, the code in util/set.py is used to initialize an object which is bound to util.set. There's a function called __import__ which takes care of all of this, and in fact the statement import util.set is basically equivalent to
util = __import__('util.set')
The point is, when you import a Python module, what you get is an object corresponding to the top-level package, util. In order to get access to util.set you need to go through that, and that's why it seems like you need to use fully qualified names in Python.
There are ways to get around this, of course. Since all these things are objects, one simple approach is to just bind util.set to a simpler name, i.e. after the import statement, you can have
set = util.set
and from that point on you can just use set where you otherwise would have written util.set. (Of course this obscures the built-in set class, so I don't recommend actually using the name set.) Or, as mentioned in at least one other answer, you could write
from util import set
or
import util.set as set
This still imports the package util with the module set in it, but instead of creating a variable util in the current scope, it creates a variable set that refers to util.set. Behind the scenes, this works kind of like
_util = __import__('util', fromlist='set')
set = _util.set
del _util
in the former case, or
_util = __import__('util.set')
set = _util.set
del _util
in the latter (although both ways do essentially the same thing). This form is semantically more like what Java's import statement does: it defines an alias (set) to something that would ordinarily only be accessible by a fully qualified name (util.set).
You can shorten it, if you would like:
import foo.bar as whateveriwant
Using the full name prevents two packages with the same-named submodules from clobbering each other.
There is a module in the standard library called io:
In [84]: import io
In [85]: io
Out[85]: <module 'io' from '/usr/lib/python2.6/io.pyc'>
There is also a module in scipy called io:
In [95]: import scipy.io
In [96]: scipy.io
Out[96]: <module 'scipy.io' from '/usr/lib/python2.6/dist-packages/scipy/io/__init__.pyc'>
If you wanted to use both modules in the same script, then namespaces are a convenient way to distinguish the two.
In [97]: import this
The Zen of Python, by Tim Peters
...
Namespaces are one honking great idea -- let's do more of those!
in Python, importing doesn't just indicate you might use something. The import actually executes code at the module level. You can think of the import as being the moment where the functions are 'interpreted' and created. Any code that is in the _____init_____.py level or not inside a function or class definition happens then.
The import also makes an inexpensive copy of the whole module's namespace and puts it inside the namespace of the file / module / whatever where it is imported. An IDE then has a list of the functions you might be starting to type for command completion.
Part of the Python philosophy is explicit is better than implicit. Python could automatically import the first time you try to access something from a package, but that's not explicit.
I'm also guessing that package initialization would be much more difficult if the imports were automatic, as it wouldn't be done consistently in the code.
You're a bit confused about how Python imports work. (I was too when I first started.) In Python, you can't simply refer to something within a module by the full name, unlike in Java; you HAVE to import the module first, regardless of how you plan on referring to the imported item. Try typing math.sqrt(5) in the interpreter without importing math or math.sqrt first and see what happens.
Anyway... the reason import foo.bar has you required to use foo.bar instead of just bar is to prevent accidental namespace conflicts. For example, what if you do import foo.bar, and then import baz.bar?
You could, of course, choose to do import foo.bar as bar (i.e. aliasing), but if you're doing that you may as well just use from foo import bar. (EDIT: except when you want to import methods and variables. Then you have to use the from ... import ... syntax. This includes instances where you want to import a method or variable without aliasing, i.e. you can't simply do import foo.bar if bar is a method or variable.)
Other than in Java, in Python import foo.bar declares, that you are going to use the thing referred to by foo.bar.
This matches with Python's philosophy that explicit is better than implicit. There are more programming languages that make inter-module dependencies more explicit than Java, for example Ada.
Using the full name makes it possible to disambiguate definitions with the same name coming from different modules.
You don't have to use the full name. Try one of these
from foo import bar
import foo.bar as bar
import foo.bar
bar = foo.bar
from foo import *
A few reasons why explicit imports are good:
They help signal to humans and tools what packages your module depends on.
They avoid the overhead of dynamically determining which packages have to be loaded (and possibly compiled) at run time.
They (along with sys.path) unambiguously distinguish symbols with conflicting names from different namespaces.
They give the programmer some control of what enters the namespace within which he is working.