If I declare an instance at the module level, which is to be set once and never changed, should I stick to uppercase naming conventions from PEP8?
E.g.
entity_manager = EntityManager(config)
vs
ENTITY_MANAGER = EntityManager(config)
There is usually no discussion when the constant is a primitive type, e.g. an integer. But what about this?
The general spirit of PEP8 is to "Keep your code consistent", whether that constant is int, string, or class instance.
According to PEP8, use upper case letters separated by underscore for constants that won't be changed through out your code. That coupled with consistency, should be applied all throughout your constants, including class instances.
On using UpperCase letters:
Constants are usually defined on a module level and written in all
capital letters with underscores separating words. Examples include
MAX_OVERFLOW and TOTAL.
I agree with the general spirit of PEP8 and to keeping your code consistent. If there are issues you have with PEP8, update your .pylintrc after reaching a consensus with your team.
Regarding the original post, I also agree with the uppercase declarations of "constants," regardless of them being primitives or objects. What I disagree with, and this goes against convention, is the use of all uppercase at the module level for everything that may be immutable. Since Python does not enforce this directly, it's up to us to define naming conventions that imply both purpose and scope for developers picking up where you left off. If I don't want my "constant" to be accessible from another module, there's not much I can do in Python other than prefix with an underscore, make lowercase, or obfuscate the name in some way.
In other languages, a general guideline that helps produce better code is always make everything as hidden as possible. If in doubt about whether a variable should be private or protected, it's better to go with private.
Does the same hold true for Python? Should I use two leading underscores on everything at first, and only make them less hidden (only one underscore) as I need them?
If the convention is to use only one underscore, I'd also like to know the rationale.
Here's a comment I left on JBernardo's answer. It explains why I asked this question and also why I'd like to know why Python is different from the other languages:
I come from languages that train you to think everything should be only as public as needed and no more. The reasoning is that this will reduce dependencies and make the code safer to alter. The Python way of doing things in reverse -- starting from public and going towards hidden -- is odd to me.
When in doubt, leave it "public" - I mean, do not add anything to obscure the name of your attribute. If you have a class with some internal value, do not bother about it. Instead of writing:
class Stack(object):
def __init__(self):
self.__storage = [] # Too uptight
def push(self, value):
self.__storage.append(value)
write this by default:
class Stack(object):
def __init__(self):
self.storage = [] # No mangling
def push(self, value):
self.storage.append(value)
This is for sure a controversial way of doing things. Python newbies hate it, and even some old Python guys despise this default - but it is the default anyway, so I recommend you to follow it, even if you feel uncomfortable.
If you really want to send the message "Can't touch this!" to your users, the usual way is to precede the variable with one underscore. This is just a convention, but people understand it and take double care when dealing with such stuff:
class Stack(object):
def __init__(self):
self._storage = [] # This is ok, but Pythonistas use it to be relaxed about it
def push(self, value):
self._storage.append(value)
This can be useful, too, for avoiding conflict between property names and attribute names:
class Person(object):
def __init__(self, name, age):
self.name = name
self._age = age if age >= 0 else 0
#property
def age(self):
return self._age
#age.setter
def age(self, age):
if age >= 0:
self._age = age
else:
self._age = 0
What about the double underscore? Well, we use the double underscore magic mainly to avoid accidental overloading of methods and name conflicts with superclasses' attributes. It can be pretty valuable if you write a class to be extended many times.
If you want to use it for other purposes, you can, but it is neither usual nor recommended.
EDIT: Why is this so? Well, the usual Python style does not emphasize making things private - on the contrary! There are many reasons for that - most of them controversial... Let us see some of them.
Python has properties
Today, most OO languages use the opposite approach: what should not be used should not be visible, so attributes should be private. Theoretically, this would yield more manageable, less coupled classes because no one would change the objects' values recklessly.
However, it is not so simple. For example, Java classes have many getters that only get the values and setters that only set the values. You need, let us say, seven lines of code to declare a single attribute - which a Python programmer would say is needlessly complex. Also, you write a lot of code to get one public field since you can change its value using the getters and setters in practice.
So why follow this private-by-default policy? Just make your attributes public by default. Of course, this is problematic in Java because if you decide to add some validation to your attribute, it would require you to change all:
person.age = age;
in your code to, let us say,
person.setAge(age);
setAge() being:
public void setAge(int age) {
if (age >= 0) {
this.age = age;
} else {
this.age = 0;
}
}
So in Java (and other languages), the default is to use getters and setters anyway because they can be annoying to write but can spare you much time if you find yourself in the situation I've described.
However, you do not need to do it in Python since Python has properties. If you have this class:
class Person(object):
def __init__(self, name, age):
self.name = name
self.age = age
...and then you decide to validate ages, you do not need to change the person.age = age pieces of your code. Just add a property (as shown below)
class Person(object):
def __init__(self, name, age):
self.name = name
self._age = age if age >= 0 else 0
#property
def age(self):
return self._age
#age.setter
def age(self, age):
if age >= 0:
self._age = age
else:
self._age = 0
Suppose you can do it and still use person.age = age, why would you add private fields and getters and setters?
(Also, see Python is not Java and this article about the harms of using getters and setters.).
Everything is visible anyway - and trying to hide complicates your work
Even in languages with private attributes, you can access them through some reflection/introspection library. And people do it a lot, in frameworks and for solving urgent needs. The problem is that introspection libraries are just a complicated way of doing what you could do with public attributes.
Since Python is a very dynamic language, adding this burden to your classes is counterproductive.
The problem is not being possible to see - it is being required to see
For a Pythonista, encapsulation is not the inability to see the internals of classes but the possibility of avoiding looking at it. Encapsulation is the property of a component that the user can use without concerning about the internal details. If you can use a component without bothering yourself about its implementation, then it is encapsulated (in the opinion of a Python programmer).
Now, if you wrote a class you can use it without thinking about implementation details, there is no problem if you want to look inside the class for some reason. The point is: your API should be good, and the rest is details.
Guido said so
Well, this is not controversial: he said so, actually. (Look for "open kimono.")
This is culture
Yes, there are some reasons, but no critical reason. This is primarily a cultural aspect of programming in Python. Frankly, it could be the other way, too - but it is not. Also, you could just as easily ask the other way around: why do some languages use private attributes by default? For the same main reason as for the Python practice: because it is the culture of these languages, and each choice has advantages and disadvantages.
Since there already is this culture, you are well-advised to follow it. Otherwise, you will get annoyed by Python programmers telling you to remove the __ from your code when you ask a question in Stack Overflow :)
First - What is name mangling?
Name mangling is invoked when you are in a class definition and use __any_name or __any_name_, that is, two (or more) leading underscores and at most one trailing underscore.
class Demo:
__any_name = "__any_name"
__any_other_name_ = "__any_other_name_"
And now:
>>> [n for n in dir(Demo) if 'any' in n]
['_Demo__any_name', '_Demo__any_other_name_']
>>> Demo._Demo__any_name
'__any_name'
>>> Demo._Demo__any_other_name_
'__any_other_name_'
When in doubt, do what?
The ostensible use is to prevent subclassers from using an attribute that the class uses.
A potential value is in avoiding name collisions with subclassers who want to override behavior, so that the parent class functionality keeps working as expected. However, the example in the Python documentation is not Liskov substitutable, and no examples come to mind where I have found this useful.
The downsides are that it increases cognitive load for reading and understanding a code base, and especially so when debugging where you see the double underscore name in the source and a mangled name in the debugger.
My personal approach is to intentionally avoid it. I work on a very large code base. The rare uses of it stick out like a sore thumb and do not seem justified.
You do need to be aware of it so you know it when you see it.
PEP 8
PEP 8, the Python standard library style guide, currently says (abridged):
There is some controversy about the use of __names.
If your class is intended to be subclassed, and you have attributes that you do not want subclasses to use, consider naming them with double leading underscores and no trailing underscores.
Note that only the simple class name is used in the mangled name, so if a subclass chooses both the same class name and attribute name,
you can still get name collisions.
Name mangling can make certain uses, such as debugging and __getattr__() , less convenient. However the name mangling algorithm is well documented and easy to perform manually.
Not everyone likes name mangling. Try to balance the need to avoid accidental name clashes with potential use by advanced callers.
How does it work?
If you prepend two underscores (without ending double-underscores) in a class definition, the name will be mangled, and an underscore followed by the class name will be prepended on the object:
>>> class Foo(object):
... __foobar = None
... _foobaz = None
... __fooquux__ = None
...
>>> [name for name in dir(Foo) if 'foo' in name]
['_Foo__foobar', '__fooquux__', '_foobaz']
Note that names will only get mangled when the class definition is parsed:
>>> Foo.__test = None
>>> Foo.__test
>>> Foo._Foo__test
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'Foo' has no attribute '_Foo__test'
Also, those new to Python sometimes have trouble understanding what's going on when they can't manually access a name they see defined in a class definition. This is not a strong reason against it, but it's something to consider if you have a learning audience.
One Underscore?
If the convention is to use only one underscore, I'd also like to know the rationale.
When my intention is for users to keep their hands off an attribute, I tend to only use the one underscore, but that's because in my mental model, subclassers would have access to the name (which they always have, as they can easily spot the mangled name anyways).
If I were reviewing code that uses the __ prefix, I would ask why they're invoking name mangling, and if they couldn't do just as well with a single underscore, keeping in mind that if subclassers choose the same names for the class and class attribute there will be a name collision in spite of this.
I wouldn't say that practice produces better code. Visibility modifiers only distract you from the task at hand, and as a side effect force your interface to be used as you intended. Generally speaking, enforcing visibility prevents programmers from messing things up if they haven't read the documentation properly.
A far better solution is the route that Python encourages: Your classes and variables should be well documented, and their behaviour clear. The source should be available. This is far more extensible and reliable way to write code.
My strategy in Python is this:
Just write the damn thing, make no assumptions about how your data should be protected. This assumes that you write to create the ideal interfaces for your problems.
Use a leading underscore for stuff that probably won't be used externally, and isn't part of the normal "client code" interface.
Use double underscore only for things that are purely convenience inside the class, or will cause considerable damage if accidentally exposed.
Above all, it should be clear what everything does. Document it if someone else will be using it. Document it if you want it to be useful in a year's time.
As a side note, you should actually be going with protected in those other languages: You never know your class might be inherited later and for what it might be used. Best to only protect those variables that you are certain cannot or should not be used by foreign code.
You shouldn't start with private data and make it public as necessary. Rather, you should start by figuring out the interface of your object. I.e. you should start by figuring out what the world sees (the public stuff) and then figure out what private stuff is necessary for that to happen.
Other language make difficult to make private that which once was public. I.e. I'll break lots of code if I make my variable private or protected. But with properties in python this isn't the case. Rather, I can maintain the same interface even with rearranging the internal data.
The difference between _ and __ is that python actually makes an attempt to enforce the latter. Of course, it doesn't try really hard but it does make it difficult. Having _ merely tells other programmers what the intention is, they are free to ignore at their peril. But ignoring that rule is sometimes helpful. Examples include debugging, temporary hacks, and working with third party code that wasn't intended to be used the way you use it.
There are already a lot of good answers to this, but I'm going to offer another one. This is also partially a response to people who keep saying that double underscore isn't private (it really is).
If you look at Java/C#, both of them have private/protected/public. All of these are compile-time constructs. They are only enforced at the time of compilation. If you were to use reflection in Java/C#, you could easily access private method.
Now every time you call a function in Python, you are inherently using reflection. These pieces of code are the same in Python.
lst = []
lst.append(1)
getattr(lst, 'append')(1)
The "dot" syntax is only syntactic sugar for the latter piece of code. Mostly because using getattr is already ugly with only one function call. It just gets worse from there.
So with that, there can't be a Java/C# version of private, as Python doesn't compile the code. Java and C# can't check if a function is private or public at runtime, as that information is gone (and it has no knowledge of where the function is being called from).
Now with that information, the name mangling of the double underscore makes the most sense for achieving "private-ness". Now when a function is called from the 'self' instance and it notices that it starts with '__', it just performs the name mangling right there. It's just more syntactic sugar. That syntactic sugar allows the equivalent of 'private' in a language that only uses reflection for data member access.
Disclaimer: I have never heard anybody from the Python development say anything like this. The real reason for the lack of "private" is cultural, but you'll also notice that most scripting/interpreted languages have no private. A strictly enforceable private is not practical at anything except for compile time.
First: Why do you want to hide your data? Why is that so important?
Most of the time you don't really want to do it but you do because others are doing.
If you really really really don't want people using something, add one underscore in front of it. That's it... Pythonistas know that things with one underscore is not guaranteed to work every time and may change without you knowing.
That's the way we live and we're okay with that.
Using two underscores will make your class so bad to subclass that even you will not want to work that way.
The chosen answer does a good job of explaining how properties remove the need for private attributes, but I would also add that functions at the module level remove the need for private methods.
If you turn a method into a function at the module level, you remove the opportunity for subclasses to override it. Moving some functionality to the module level is more Pythonic than trying to hide methods with name mangling.
Following code snippet will explain all different cases :
two leading underscores (__a)
single leading underscore (_a)
no underscore (a)
class Test:
def __init__(self):
self.__a = 'test1'
self._a = 'test2'
self.a = 'test3'
def change_value(self,value):
self.__a = value
return self.__a
printing all valid attributes of Test Object
testObj1 = Test()
valid_attributes = dir(testObj1)
print valid_attributes
['_Test__a', '__doc__', '__init__', '__module__', '_a', 'a',
'change_value']
Here, you can see that name of __a has been changed to _Test__a to prevent this variable to be overridden by any of the subclass. This concept is known as "Name Mangling" in python.
You can access this like this :
testObj2 = Test()
print testObj2._Test__a
test1
Similarly, in case of _a, the variable is just to notify the developer that it should be used as internal variable of that class, the python interpreter won't do anything even if you access it, but it is not a good practise.
testObj3 = Test()
print testObj3._a
test2
a variable can be accesses from anywhere it's like a public class variable.
testObj4 = Test()
print testObj4.a
test3
Hope the answer helped you :)
At first glance it should be the same as for other languages (under "other" I mean Java or C++), but it isn't.
In Java you made private all variables that shouldn't be accessible outside. In the same time in Python you can't achieve this since there is no "privateness" (as one of Python principles says - "We're all adults"). So double underscore means only "Guys, do not use this field directly". The same meaning has singe underscore, which in the same time doesn't cause any headache when you have to inherit from considered class (just an example of possible problem caused by double underscore).
So, I'd recommend you to use single underscore by default for "private" members.
"If in doubt about whether a variable should be private or protected, it's better to go with private." - yes, same holds in Python.
Some answers here say about 'conventions', but don't give the links to those conventions. The authoritative guide for Python, PEP 8 states explicitly:
If in doubt, choose non-public; it's easier to make it public later than to make a public attribute non-public.
The distinction between public and private, and name mangling in Python have been considered in other answers. From the same link,
We don't use the term "private" here, since no attribute is really private in Python (without a generally unnecessary amount of work).
#EXAMPLE PROGRAM FOR Python name mangling
class Demo:
__any_name = "__any_name"
__any_other_name_ = "__any_other_name_"
[n for n in dir(Demo) if 'any' in n] # GIVES OUTPUT AS ['_Demo__any_name',
# '_Demo__any_other_name_']
Often when I see class definitions class Foo:, I always see them start with upper case letters.
However, isn't a list [] or a dict {} or some other built-in type, a class as well? For that matter, everything typed into the Python's IDLE which is a keyword that is automatically color coded in purple (with the Window's binary distribution), is itself a class, right?
Such as spam = list()
spam is now an instance of a list()
So my question is, why does Python allow us to first of all do something like list = list() when nobody, probably, does that. But also, why is it not list = List()
Did the developers of the language decide not to use any sort of convention, while it is the case that most Python programmers do name their classes as such?
Yes, uppercase-initial classes are the convention, as outlined in PEP 8.
You are correct that many builtin types do not follow this convention. These are holdovers from earlier stages of Python when there was a much bigger difference between user-defined classes and builtin types. However, it still seems that builtin or extension types written in C are more likely to have lowercase names (e.g., numpy.array, not numpy.Array).
Nonetheless, the convention is to use uppercase-initial for your own classes in Python code.
PEP8 is the place to go for code style.
To address your question on why list = list() is valid, list is simply a name in the global namespace, and can be overriden like any other variable.
All are in the end design decisions.
[...] why does python allow us to first of all do something like list = list()
https://www.python.org/dev/peps/pep-0020/ says (try also >>> import this)
Simple is better than complex.
Special cases aren't special enough to break the rules.
And this would be a special case.
[...] why is it not list = List()
https://www.python.org/dev/peps/pep-0008/#class-names says:
Class Names
Class names should normally use the CapWords convention.
The naming convention for functions may be used instead in cases where
the interface is documented and used primarily as a callable.
Note that there is a separate convention for builtin names: most
builtin names are single words (or two words run together), with the
CapWords convention used only for exception names and builtin
constants. [emphasis mine]
All other classes should use the CapWorlds convention. As list, object, etc are built-in names, they follow this separate convention.
I think that the only person who really knows the entire answer to your question is the BDFL. Convention outside of the types implemented in C is definitely to use upper-case (as detailed in PEP8). However, it's interesting to note that not all C-level types follow the convention (i.e. Py_True, Py_False) do not. Yes, they're constants at Python-level, but they're all PyTypeObjects. I'd be curious to know if that's the only distinction and, hence, the difference in convention.
According to PEP 8, a nice set of guidelines for Python developers:
Almost without exception, class names use the CapWords convention.
Classes for internal use have a leading underscore in addition.
PEP 8 is directed at Python development for the standard library in the main Python distribution, but they're sensible guidelines to follow.
I'm new to Python. I'm writing some code in Sublime and it highlights the word 'input'
I use it as a variable name and it seems to work, so I wondered whether it may be a keyword in a newer version. (I'm currently using 2.7.5)
No, input is not a keyword. Instead, it is a built-in function.
And yes, you can create a variable with the name input. But please don't. Doing so is a bad practice because it overshadows the built-in (makes it unusable in the current scope).
If you must use the name input, the convention is to place an underscore after it:
input_ = input()
input is not a keyword, it's a function provided by the standard library and included in the builtins module (this module provides globally accessible variables and functions.):
>>> import builtins
>>> input is builtins.input
True
And sure, you can create a variable with the name input. It's perfectly fine for experienced and intermediate users to do so because they can easily figure that the input name has been re-used.
Use the best name for the content/intent you want to convey. If input is the best then use it (provided you don't need the builtin), and don't confuse readers with names like input_ (beginners will wonder whether there's a special meaning to a trailing underscore)
But if you're a beginner please don't re-define builtins, by overshadowing the built-in input (overshadowing a variable makes it unusable in the current scope) you'll end-up with this error when calling input() later on (in the same scope) and you may struggle to figure out why:
TypeError: 'str' object is not callable
Beginners should instead use another name, preferably not input_ because underscores have special meanings in python, as a result other beginners will wonder whether there's a special meaning for that trailing underscore (is it the same or related to leading underscores? or maybe to double underscores?)
In another comment someone stated that it is a bad practice to overshadow variables and he even came up with a convention that he borrowed from another use. After all, if overshadowing variables were a really bad practice, the python language designers wouldn't have allowed it in the first place, but they know and recognize that it has the potential to improve readability, just as it does in other languages. So they allowed it, and it also ease transition to Python from other languages where overshadowing is also allowed like C/C++, Java and even bash.
note: the conventional use for a trailing underscore is where it's impossible to use a name, like the keyword class in Python. Then you'd use class_ (but like I wrote above, it's best to avoid it in Python because underscores can confuse beginners as they can convey special meanings)
I have noticed that in Python some of the fundamental types have two types of methods: those that are surrounded by __ and those that don't.
For example, if I have a variable of type float called my_number, I can see in IPython that it has the following methods:
my_number.__abs__ my_number.__pos__
my_number.__add__ my_number.__pow__
my_number.__class__ my_number.__radd__
my_number.__coerce__ my_number.__rdiv__
my_number.__delattr__ my_number.__rdivmod__
my_number.__div__ my_number.__reduce__
my_number.__divmod__ my_number.__reduce_ex__
my_number.__doc__ my_number.__repr__
my_number.__eq__ my_number.__rfloordiv__
my_number.__float__ my_number.__rmod__
my_number.__floordiv__ my_number.__rmul__
my_number.__format__ my_number.__rpow__
my_number.__ge__ my_number.__rsub__
my_number.__getattribute__ my_number.__rtruediv__
my_number.__getformat__ my_number.__setattr__
my_number.__getnewargs__ my_number.__setformat__
my_number.__gt__ my_number.__sizeof__
my_number.__hash__ my_number.__str__
my_number.__init__ my_number.__sub__
my_number.__int__ my_number.__subclasshook__
my_number.__le__ my_number.__truediv__
my_number.__long__ my_number.__trunc__
my_number.__lt__ my_number.as_integer_ratio
my_number.__mod__ my_number.conjugate
my_number.__mul__ my_number.fromhex
my_number.__ne__ my_number.hex
my_number.__neg__ my_number.imag
my_number.__new__ my_number.is_integer
my_number.__nonzero__ my_number.real
What is the difference between those that are surrounded by ___ and those that aren't?
Is this some sort of standard used in other programming languages? Does it usually mean the same thing in similar languages?
Generally, "double underscore" methods are used internally by Python for certain builtin functions or operators (e.g. __add__ defines behavior for the +). Those that do not have double underscores are just normal methods that wouldn't be used by operators or builtins. Now, these methods are still "normal" methods, in that you can call them just like any other method, but parts of the Python core treat them specially.
No, as far as I am aware, this is unique to Python, though many other languages support a similar idea (builtin/operator overloading) but through different mechanisms.
Shameless plug: I wrote a guide to this aspect of Python last year which is fairly comprehensive, you can read about how to use these methods on your own objects at http://rafekettler.com/magicmethods.html.
1) The variables you speak of __*__ are system variables or methods.
To quote the Python reference guide:
System-defined names. These names are defined by the interpreter and its implementation (including the standard library); applications should not expect to define additional names using this convention. The set of names of this class defined by Python may be extended in future versions.
Essentially they are variables or methods pre defined by the system. For example the system variable __name__ can be used within any function and will always contain the name of that function. You can find more comprehensive information and examples here.
2) This concept of system reserved variables is fundamental in most programming languages. For example PHP refers to them as Magic Constants. The Python example above to get a function name can be achieved in PHP using __FUNCTION__. More examples here.
From documentation:
Certain classes of identifiers (besides keywords) have special
meanings. These classes are identified by the patterns of leading and
trailing underscore characters:
_*
Not imported by from module import *. The special identifier _ is used in the interactive interpreter to store the result of the last
evaluation; it is stored in the builtin module. When not in
interactive mode, _ has no special meaning and is not defined. See
section The import statement.
Note The name _ is often used in conjunction with internationalization; refer to the documentation for the gettext
module for more information on this convention.
__ * __
System-defined names. These names are defined by the interpreter and its implementation (including the standard library). Current
system names are discussed in the Special method names section and
elsewhere. More will likely be defined in future versions of Python.
Any use of * names, in any context, that does not follow
explicitly documented use, is subject to breakage without warning.
__*
Class-private names. Names in this category, when used within the context of a class definition, are re-written to use a mangled form to
help avoid name clashes between “private” attributes of base and
derived classes. See section Identifiers (Names).
http://docs.python.org/reference/lexical_analysis.html#reserved-classes-of-identifiers
These are special methods that are called in specific contexts.
Read the documentation for the details.:
A class can implement certain operations that are invoked by special
syntax (such as arithmetic operations or subscripting and slicing) by
defining methods with special names. This is Python’s approach to
operator overloading, allowing classes to define their own behavior
with respect to language operators. For instance, if a class defines a
method named __getitem__(), and x is an instance of this class, then
x[i] is roughly equivalent to x.__getitem__(i) for old-style classes
and type(x).__getitem__(x, i) for new-style classes. Except where
mentioned, attempts to execute an operation raise an exception when no
appropriate method is defined (typically AttributeError or TypeError).
To answer your second question, this convention is used in C with macros such as __FILE__. Guido van Rossum, the creator of Python, explicitly says this was his inspiration in his blog on the history of Python.
its used to indicate that the attributes are internal. there is no actual privacy in python, so these are used as hints yo indicate internals rather than api methods.