Using pydoc for non-public/private classes - python

When I use _ as a prefix for a class name (see also this and this), the class becomes non-public and is not displayed by pydoc:
class _DoNotShowMeInPydoc:
"""Class Documentation which is not displayed"""
That's just what I want, because this class is used only internally and is of no importance for people who use the module.
But - what if someone comes along who wants to know about the internals of my module, which are (of course) also documented. Is there a flag or an option for pydoc to display the non-public classes?

Let's just assume there are a variety of audiences for documentation and there is value in reading code through documentation separate from the source code itself.
Pydoc supports almost no switches to control the output. Access levels are generally ad hoc conventions and not part of the python language.
Pydoc follows this not so easy to summarize rules to decide what is private-as-in-do-not-document. So your only option is to change your code to get it passed the visiblename() function.
Pdoc3, a pydoc replacement, uses the __pdoc__ dictionary to override rules that are similar to pydoc's.
I created a fork of pydoc that generates documentation for everything by default.

Related

Python/Django and services as classes

Are there any conventions on how to implement services in Django? Coming from a Java background, we create services for business logic and we "inject" them wherever we need them.
Not sure if I'm using python/django the wrong way, but I need to connect to a 3rd party API, so I'm using an api_service.py file to do that. The question is, I want to define this service as a class, and in Java, I can inject this class wherever I need it and it acts more or less like a singleton. Is there something like this I can use with Django or should I build the service as a singleton and get the instance somewhere or even have just separate functions and no classes?
TL;DR It's hard to tell without more details but chances are you only need a mere module with a couple plain functions or at most just a couple simple classes.
Longest answer:
Python is not Java. You can of course (technically I mean) use Java-ish designs, but this is usually not the best thing to do.
Your description of the problem to solve is a bit too vague to come with a concrete answer, but we can at least give you a few hints and pointers (no pun intended):
1/ Everything is an object
In python, everything (well, everything you can find on the RHS of an assignment that is) is an object, including modules, classes, functions and methods.
One of the consequences is that you don't need any complex framework for dependency injection - you just pass the desired object (module, class, function, method, whatever) as argument and you're done.
Another consequence is that you don't necessarily need classes for everything - a plain function or module can be just enough.
A typical use case is the strategy pattern, which, in Python, is most often implemented using a mere callback function (or any other callable FWIW).
2/ a python module is a singleton.
As stated above, at runtime a python module is an object (of type module) whose attributes are the names defined at the module's top-level.
Except for some (pathological) corner cases, a python module is only imported once for a given process and is garanteed to be unique. Combined with the fact that python's "global" scope is really only "module-level" global, this make modules proper singletons, so this design pattern is actually already builtin.
3/ a python class is (almost) a singleton
Python classes are objects too (instance of type type, directly or indirectly), and python has classmethods (methods that act on the class itself instead of acting on the current instance) and class-level attributes (attributes that belong to the class object itself, not to it's instances), so if you write a class that only has classmethods and class attributes, you technically have a singleton - and you can use this class either directly or thru instances without any difference since classmethods can be called on instances too.
The main difference here wrt/ "modules as singletons" is that with classes you can use inheritance...
4/ python has callables
Python has the concept of "callable" objects. A "callable" is an object whose class implements the __call__() operator), and each such object can be called as if it was a function.
This means that you can not only use functions as objects but also use objects as functions - IOW, the "functor" pattern is builtin. This makes it very easy to "capture" some context in one part of the code and use this context for computations in another part.
5/ a python class is a factory
Python has no new keyword. Pythonc classes are callables, and instanciation is done by just calling the class.
This means that you can actually use a class or function the same way to get an instance, so the "factory" pattern is also builtin.
6/ python has computed attributes
and beside the most obvious application (replacing a public attribute by a pair of getter/setter without breaking client code), this - combined with other features like callables etc - can prove to be very powerful. As a matter of fact, that's how functions defined in a class become methods
7/ Python is dynamic
Python's objects are (usually) dict-based (there are exceptions but those are few and mostly low-level C-coded classes), which means you can dynamically add / replace (and even remove) attributes and methods (since methods are attributes) on a per-instance or per-class basis.
While this is not a feature you want to use without reasons, it's still a very powerful one as it allows to dynamically customize an object (remember that classes are objects too), allowing for more complex objects and classes creation schemes than what you can do in a static language.
But Python's dynamic nature goes even further - you can use class decorators and/or metaclasses to taylor the creation of a class object (you may want to have a look at Django models source code for a concrete example), or even just dynamically create a new class using it's metaclass and a dict of functions and other class-level attributes.
Here again, this can really make seemingly complex issues a breeze to solve (and avoid a lot of boilerplate code).
Actually, Python exposes and lets you hook into most of it's inners (object model, attribute resolution rules, import mechanism etc), so once you understand the whole design and how everything fits together you really have the hand on most aspects of your code at runtime.
Python is not Java
Now I understand that all of this looks a bit like a vendor's catalog, but the point is highlight how Python differs from Java and why canonical Java solutions - or (at least) canonical Java implementations of those solutions - usually don't port well to the Python world. It's not that they don't work at all, just that Python usually has more straightforward (and much simpler IMHO) ways to implement common (and less common) design patterns.
wrt/ your concrete use case, you will have to post a much more detailed description, but "connecting to a 3rd part API" (I assume a REST api ?) from a Django project is so trivial that it really doesn't warrant much design considerations by itself.
In Python you can write the same as Java program structure. You don't need to be so strongly typed but you can. I'm using types when creating common classes and libraries that are used across multiple scripts.
Here you can read about Python typing
You can do the same here in Python. Define your class in package (folder) called services
Then if you want singleton you can do like that:
class Service(object):
instance = None
def __new__(cls):
if cls.instance is not None:
return cls.instance
else:
inst = cls.instance = super(Service, cls).__new__()
return inst
And now you import it wherever you want in the rest of the code
from services import Service
Service().do_action()
Adding to the answer given by bruno desthuilliers and TreantBG.
There are certain questions that you can ask about the requirements.
For example one question could be, does the api being called change with different type of objects ?
If the api doesn't change, you will probably be okay with keeping it as a method in some file or class.
If it does change, such that you are calling API 1 for some scenario, API 2 for some and so on and so forth, you will likely be better off with moving/abstracting this logic out to some class (from a better code organisation point of view).
PS: Python allows you to be as flexible as you want when it comes to code organisation. It's really upto you to decide on how you want to organise the code.

How can i know what parameters i can pass to a python function and what is the proper values?

For example, when i use matplotlib as plt, a possible statement is like below:
plt.plot(x,y,color='blue')
so how can i get what arguments like 'color' i can pass to the 'plot' function, and what is the proper values for that argument?
Especially when i use some modules.
thanks for any answers.
I'm a little disappointed that this post is being downvoted because I think it's a very legitimate question. In particular, I appreciate that you asked not what the answer was but instead how you could find it for yourself in the future.
Exploring Local Python Documentation
Python has a very robust built-in documentation system as well as a very active and supportive community. At any point in time, you can use the help function to examine a particular object that you want more information on - this will pull up the documentation for that object. In your example, you could do something like:
help(plt.plot)
to pull up the documentation for the matplotlib.pyplot.plot function. Outside of a running Python process, you can use the pydoc command line tool to read and explore that same documentation. Something like:
$ pydoc matplotlib.pyplot.plot
Running that in the shell will display the same documentation as the help command example.
Writing Documentation
As a good citizen of the Python ecosystem, you'll naturally want to document your own code. This is done pretty simply by adding a docstring to the top of a function, class, or module. Docstrings are denoted with a triple quotation mark """, seen in the examples below:
"""This module contains some example classes and functions"""
class MyClass(object):
"""MyClass does some things"""
pass
def my_function(a):
"""Calculates the sum"""
return(a)
There are many different documentation styles that people prefer, so what you choose is up to you. Though I would recommend the official docstring standards outlined in PEP 257.
Finding Online Resources
It's also often useful to take advantage of online documentation and resources. The official Python documentation includes all of the builtin documentation for the standard libraries as well as a tutorial for developers who are new to Python!
As it seems that you're relatively new to the ecosystem yourself, here's some more resources that you might find useful:
Learn Python the Hard Way
CodeAcademy has a Python track
StackOverflow, obviously

Python double leading and trailing underscore usage for class attributes

In regards to naming conventions for class attributes, PEP 8 states:
__double_leading_and_trailing_underscore__: "magic" objects or attributes that live in user-controlled namespaces. E.g. __init__, __import__ or __file__. Never invent such names; only use them as documented.
Does this imply to never use this convention or only use in a "user-controlled namespace."
I have seen this used in other libraries not directly included within Python a few times. From what I gather, using this convention is more specific to implementing an API.
A couple of examples would be providing an __acl__ to a class in Pyramid, or adding __tablename__ to a class in SQLAlchemy.
Is using this convention in the context of an API OK, or should it only ever be used / reserved for Python?
Pyramid and SQLAlchemy have disobeyed the instruction not to invent such names.
It's not clear to me what you mean by "using this convention in the context of an API", but once they've invented the name, you don't have much choice but to use it as documented by them.
There's no difference between you inventing such names, and inventing them only in user-controlled namespaces. Since you're a user of Python, any namespace that you could put the name into is user-controlled. If you're modifying the Python source to add a new extension to the language that requires some "magic" name, then you can invent one. I expect if you're doing this, you'll usually be in communication with GvR one way or another, so you can ask his opinion directly :-)
What's happening here is that the library authors want a name that none of their users will use by accident to mean something else. The Python language also wants names that no user will use by accident to mean something else, so it "reserves" names of a particular format for use by the language. But then the library-writer has decided to "steal" one of those reserved names and use it anyway, because they feel that's the best way to avoid a clash with one of their users. They have tens of thousands of users, most of whom they don't know anything about. But there's only one Python language spec, and they have access to it. So if a clash occurs the library developers will know about it, which is the plus side, but it will be their fault and difficult to fix, which is the minus side.
Perhaps they're hoping that by using it, the have de facto reserved it for themselves, and that GvR will choose never to use a name that a popular library has already used. Or perhaps they've discussed it on the relevant mailing lists and obtained an exception to the usual rule -- I don't know whether there's a process for that.

Documenting a non-existing member with Doxygen

I'm trying to document a python class using Doxygen. The class exposes a set of properties over d-bus, but these have no corresponding public getters/setters in the python class. Instead, they are implemented through a d-bus properties interface (Set/Get/GetAll/Introspect).
What I want to do is to be able to document these properties using something like this:
## #property package::Class::Name description
The whole package::Class works (the same method finds functions, so it finds the right class).
When running doxygen I get the following error:
warning: documented function ``package::Class::Name' was not declared or defined.
I can live with a warning, but unfortunately the property fails to appear in the documentation generated for the class, so it is not only a warning, but it is silenced as well.
So, my question is, how, if I can, do I make the non-existing property member appear in the generated docs?
Define the attribute inside an if 0: block:
## #class X
## #brief this is useless
class X:
if 0:
## #brief whatevs is a property that doesn't exist in spacetime
##
## It is designed to make bunny cry.
whatevs = property
This will cause it to exist in the documentation (tested with doxygen 1.8.1.2-1 on debian-squeeze). The attribute will never be made to exist at runtime, and in fact it looks like the python bytecode optimizer eliminates if statement and its body altogether.
I looked into something similar previously and couldn't find a direct way to coax Doxygen into documenting an undefined member. There are two basic kludges you can use here:
1.) generate a dummy object (or dummy members) for doxygen to inventory which don't actually exist in the live code.
2.) If the adjustments you need are fairly predictable and regular you could write an INPUT_FILTER for doxygen which takes your files and converts them before parsing. There are some issues with this method--mostly that if you plan on including the code in the documentation and the filter has to add/remove lines from the file, the line numbers it indicates will be off, and any code windows shown with the documentation will be off by that number of lines. You can also check the option to filter the displayed sources to adjust for this, but depending on who the consumer of your documentation is, it may be confusing for the copy in Doxygen not to perfectly match what's in the real source.
In our case we use a python script which Doxygen runs from the command-line with the file path as the arg. We read the file indicated and write what we want Doxygen to interpret instead to stdout. If you need the source copies displayed in doxygen to be filtered as well you can set FILTER_SOURCE_FILES to YES.

What is the underscore prefix for python file name?

In cherryPy for example, there are files like:
__init__.py
_cptools.py
How are they different? What does this mean?
__...__ means reserved Python name (both in filenames and in other names). You shouldn't invent your own names using the double-underscore notation; and if you use existing, they have special functionality.
In this particular example, __init__.py defines the 'main' unit for a package; it also causes Python to treat the specific directory as a package. It is the unit that will be used when you call import cherryPy (and cherryPy is a directory). This is briefly explained in the Modules tutorial.
Another example is the __eq__ method which provides equality comparison for a class. You are allowed to call those methods directly (and you use them implicitly when you use the == operator, for example); however, newer Python versions may define more such methods and thus you shouldn't invent your own __-names because they might then collide. You can find quite a detailed list of such methods in Data model docs.
_... is often used as 'internal' name. For example, modules starting with _ shouldn't be used directly; similarly, methods with _ are supposedly-private and so on. It's just a convention but you should respect it.
These, and other, naming conventions are described in detail in Style Guide for Python Code - Descriptive: Naming Styles
Briefly:
__double_leading_and_trailing_underscore__: "magic" objects or attributes that live in user-controlled namespaces. E.g.__init__, __import__ or __file__. Never invent such names; only use them as documented.
_single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose name starts with an underscore.
__init__.py is a special file that, when existing in a folder turns that folder into module. Upon importing the module, __init__.py gets executed. The other one is just a naming convention but I would guess this would say that you shouldn't import that file directly.
Take a look here: 6.4. Packages for an explanation of how to create modules.
General rule: If anything in Python is namend __anything__ then it is something special and you should read about it before using it (e.g. magic functions).
The current chosen answer already gave good explanation on the double-underscore notation for __init__.py.
And I believe there is no real need for _cptools.py notation in a filename. It is presumably an unnecessary extended usage of applying the "single leading underscore" rule from the Style Guide for Python Code - Descriptive: Naming Styles:
_single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose name starts with an underscore.
If anything, the said Style Guide actually is against using _single_leading_underscore.py in filename. Its Package and Module Names section only mentions such usage when a module is implemented in C/C++.
In general, that _single_leading_underscore notation is typically observed in function names, method names and member variables, to differentiate them from other normal methods.
There is few need (if any at all), to use _single_leading_underscore.py on filename, because the developers are not scrapers , they are unlikely to salvage a file based on its filename. They would just follow a package's highest level of APIs (technically speaking, its exposed entities defined by __all__), therefore all the filenames are not even noticeable, let alone to be a factor, of whether a file (i.e. module) would be used.

Categories

Resources