I have a written a program in Python 3 and are using Sphinx to document it. Sphinx's autodoc is great, however it only works with Python 2. Some modules work fine in autodoc, however modules don't. Some examples: Python 2 complains about Python 3 style metaclasses, and some modules which don't exist anymore in Python 2 such as configparser. This is annoying as it cannot import the docstrings from that file.
I don't want to rewrite the whole program in Python 2, however I want to use autodoc.
One idea I had was a small program that read each Python file and removed all functionality but just left the basic function and classes with their docstrings (because autodoc imports each module and reads the docstring of a specific function or class).
import configparser
import os
class TestClass:
"""
I am a class docstring.
"""
def method(self, argument):
"""
I am a method docstring.
"""
#Some code here
print(os.getcwd())
def TestFunction():
"""
I am a function docstring.
"""
#Some more useless code here
return os.path.join("foo", "bar")
into...
class TestClass:
"""
I am a class docstring.
"""
def method(self, argument):
"""
I am a method docstring.
"""
pass
def TestFunction():
"""
I am a function docstring.
"""
pass
In this way the processed code can be read by autodoc, but still have the docstrings which is what I really need. Is this the best way of going about this, and does anyone have any suggestions as to how to write the little program which converts the code.
I can remove the metaclass problem very easily with some regular expressions, but I am struggling with the rest.
m = re.search("\(metaclass=.*\)", file_content)
if m:
file_content = "".join(file_content[:m.start()], file_content[m.end():])
Would the ast module be useful?
Thanks.
You can just install the development version of sphinx, which supports python 3.
pip-3.2 install hg+https://bitbucket.org/birkenfeld/sphinx
I tested the autodoc feature on your class and it worked.
What tends to be the solution is sprinkling try/except clauses in your code.
Python 2.6 has configparser but it is known as ConfigParser (python 3 changed the camelcase names to all lower case)
so something like:
try:
import configparser
except ImportError:
#we are in 2.x
import ConfigParser as configparser
you might want to do a few things like this where it's broken. between the two. metaclasses though between the two I'm not sure to handle.
There's a 3to2 library that can convert Python 3 code to python 2. You could try this in conjunction with Sphinx.
Related
I have a product for a plone site with a module containing a utility class, and in the module that will/should use this utility, I am trying to have it setup at the module level.
In the module containing the utility (my.product.testutility), I have this:
from five import grok
from zope.interface import Interface
class ITestUtil(Interface):
"""Interface of utility
"""
def returnTest(self):
"""return a string for now
"""
class TestUtil(object):
"""Utility class test
"""
grok.implements(ITestUtil)
def returnTest(self):
return "testing"
grok.global_utility(TestUtil, name="TestUtility")
In the module that will use this utility (my.product.stringtesting):
from five import grok
from my.package.testutility import ITestUtil
from zope import component, schema
from Products.CMFCore.interfaces import ISiteRoot
utilForTesting = component.getUtility(ITestUtil, name="TestUtility")
class IStringTest(Interface):
......
class View(grok.View):
def returnStringForTest(self):
return utilForTesting.returnTest()
I also had the template file that would call the returnStringForTest to display the string on the rendered page.
I end up getting this error unfortunately:
ComponentLookupError: (< InterfaceClass my.product.testutility.ITestUtil >, "TestUtility")
I did try several different things like using grok.GlobalUtility as a base as opposed to making it an object registering it through grok.global_utility. I did remove the name parameter in the class using this while testing this.
The documentation I was trying to follow was the References on the grok site, looking at the directives page where it has the global utility information.
Also, I am using grok 0.9.
Edit:
The version of Plone I am using is Plone 4 and the version of python I am using is 2.7.
Is it possible to have the utility set up at the module level like I was trying?
You can do what you want without relying on Zope at all.
You can change the line in my.product.stringtesting:
from
utilForTesting = component.getUtility(ITestUtil, name="TestUtility")
to
utilForTesting = TestUtil()
I have a module I am using which uses RealClass, so it is an internal dependency I don't have access to.
I want to be able to create a FakeClass which replaces the functionality of the RealClass for testing. I don't want to replace individual methods but the entire class.
I looked at stubble which seems to be what I want but I was wondering if mox or any of the other mocking frameworks have this functionality? Or what would you suggest to use? Maybe fudge, monkey-patching? Just looking for best practices with this stuff. Also any useful examples would be awesome.
Pseudo code:
from module import RealClass
class FakeClass
methodsFromRealClassOverridden
class Test(unittest.TestCase):
setup()
teardown()
test1()
stub(RealClass, FakeClass) // something like this, but really just want the functionality
classThatUsesRealClass // now will use FakeClass
UPDATE:
Here's one way I found to do it. It isn't perfect but it works.
Example:
fake = FakeClass()
stub = stubout.StubOutForTesting()
stub.Set(RealClass, 'method_1', fake.method_1)
stub.Set(RealClass, 'method_2', fake.method_2)
I think you want opinions/experiences so I'm just giving my 2 cents.
As you noticed there are a few Python testing tools/classes/frameworks, but most of the time given the simplicity/dynamism/openness of Python you will limit yourself to using ad-hoc relevant test cases which involve stubbing at the interface level, and a bit of unittest... until you start using the frameworks.
There is nothing pejorative about monkey-patching, especially when it comes to performing testing/stubbing:
#!/usr/bin/env python
# minimal example of library code
class Class:
""" a class """
def method(self, arg):
""" a method that does real work """
print("pouet %s" % arg)
#!/usr/bin/env python
# minimal example for stub and tests, overriding/wrapping one method
from Class import Class
Class._real_method = Class.method
def mymethod(self, arg):
# do what you want
print("called stub")
# in case you want to call the real function...
self._real_method(arg)
Class.method = mymethod
# ...
e = Class()
e.method("pouet")
Namespaces will allow you to patch stuff inside of imported modules inside of imported modules...
Note that the above method does not work with classes in C modules.
For them you can use a wrapper class that filters on class member names using getattr/setattr, and returns the redefined members from the wrapper class.
#!/usr/bin/env python
# Stupid minimal example replacing the sys module
# (not very useful / optimal, it's just an example of patching)
import sys
class SysWrap():
real = sys
def __getattr__(self, attr):
if attr == 'stderr':
class StdErr():
def write(self, txt):
print("[err: %s]" % txt)
return StdErr()
print("Getattr %s" % attr)
return getattr(SysWrap.real, attr)
sys = SysWrap()
# use the real stdout
sys.stdout.write("pouet")
# use fake stderr
sys.stderr.write("pouet")
Once you are becoming tired of performing ad-hoc testing, you'll find higher level stuff such as the ones you mentioned (stubble, fudge) useful, but to enjoy them and use them efficiently you have to first see the problems they solve and accept all the automatic stuff they do under the hood.
It is probable that a part of ad-hoc monkey patching will remain, it's just easier to understand, and all the tools have some limitations.
Tools empower you but you have to deeply understand them to use them efficiently.
An important aspect when deciding whether to use a tool or not is that when you transmit a chunk of code, you transmit the whole environment (including testing tools).
The next guy might not be as smart as you and skip the testing because your testing tool is too complex for him.
Generally you want to avoid using a lot of dependencies in your software.
In the end, I think that nobody will bother you if you just use unittest and ad-hoc tests/monkey-patching, provided your stuff works.
Your code might not be that complex anyway.
I am doing some heavy commandline stuff (not really web based) and am new to Python, so I was wondering how to set up my files/folders/etc. Are there "header" files where I can keep all the DB connection stuff?
How/where do I define classes and objects?
Just to give you an example of a typical Python module's source, here's something with some explanation. This is a file named "Dims.py". This is not the whole file, just some parts to give an idea what's going on.
#!/usr/bin/env python
This is the standard first line telling the shell how to execute this file. Saying /usr/bin/env python instead of /usr/bin/python tells the shell to find Python via the user's PATH; the desired Python may well be in ~/bin or /usr/local/bin.
"""Library for dealing with lengths and locations."""
If the first thing in the file is a string, it is the docstring for the module. A docstring is a string that appears immediately after the start of an item, which can be accessed via its __doc__ property. In this case, since it is the module's docstring, if a user imports this file with import Dims, then Dims.__doc__ will return this string.
# Units
MM_BASIC = 1500000
MILS_BASIC = 38100
IN_BASIC = MILS_BASIC * 1000
There are a lot of good guidelines for formatting and naming conventions in a document known as PEP (Python Enhancement Proposal) 8. These are module-level variables (constants, really) so they are written in all caps with underscores. No, I don't follow all the rules; old habits die hard. Since you're starting fresh, follow PEP 8 unless you can't.
_SCALING = 1
_SCALES = {
mm_basic: MM_BASIC,
"mm": MM_BASIC,
mils_basic: MILS_BASIC,
"mil": MILS_BASIC,
"mils": MILS_BASIC,
"basic": 1,
1: 1
}
These module-level variables have leading underscores in their names. This gives them a limited amount of "privacy", in that import Dims will not let you access Dims._SCALING. However, if you need to mess with it, you can explicitly say something like import Dims._SCALING as scaling.
def UnitsToScale(units=None):
"""Scales the given units to the current scaling."""
if units is None:
return _SCALING
elif units not in _SCALES:
raise ValueError("unrecognized units: '%s'." % units)
return _SCALES[units]
UnitsToScale is a module-level function. Note the docstring and the use of default values and exceptions. No spaces around the = in default value declarations.
class Length(object):
"""A length. Makes unit conversions easier.
The basic, mm, and mils properties can be used to get or set the length
in the desired units.
>>> x = Length(mils=1000)
>>> x.mils
1000.0
>>> x.mm
25.399999999999999
>>> x.basic
38100000L
>>> x.mils = 100
>>> x.mm
2.54
"""
The class declaration. Note the docstring has things in it that look like Python command line commands. These care called doctests, in that they are test code in the docstring. More on this later.
def __init__(self, unscaled=0, basic=None, mm=None, mils=None, units=None):
"""Constructs a Length.
Default contructor creates a length of 0.
>>> Length()
Length(basic=0)
Length(<float>) or Length(<string>) creates a length with the given
value at the current scale factor.
>>> Length(1500)
Length(basic=1500)
>>> Length("1500")
Length(basic=1500)
"""
# Straight copy
if isinstance(unscaled, Length):
self._x = unscaled._x
return
# rest omitted
This is the initializer. Unlike C++, you only get one, but you can use default arguments to make it look like several different constructors are available.
def _GetBasic(self): return self._x
def _SetBasic(self, x): self._x = x
basic = property(_GetBasic, _SetBasic, doc="""
This returns the length in basic units.""")
This is a property. It allows you to have getter/setter functions while using the same syntax as you would for accessing any other data member, in this case, myLength.basic = 10 does the same thing as myLength._SetBasic(10). Because you can do this, you should not write getter/setter functions for your data members by default. Just operate directly on the data members. If you need to have getter/setter functions later, you can convert the data member to a property and your module's users won't need to change their code. Note that the docstring is on the property, not the getter/setter functions.
If you have a property that is read-only, you can use property as a decorator to declare it. For example, if the above property was to be read-only, I would write:
#property
def basic(self):
"""This returns the length in basic units."""
return self._x
Note that the name of the property is the name of the getter method. You can also use decorators to declare setter methods in Python 2.6 or later.
def __mul__(self, other):
"""Multiplies a Length by a scalar.
>>> Length(10)*10
Length(basic=100)
>>> 10*Length(10)
Length(basic=100)
"""
if type(other) not in _NumericTypes:
return NotImplemented
return Length(basic=self._x * other)
This overrides the * operator. Note that you can return the special value NotImplemented to tell Python that this operation isn't implemented (in this case, if you try to multiply by a non-numeric type like a string).
__rmul__ = __mul__
Since code is just a value like anything else, you can assign the code of one method to another. This line tells Python that the something * Length operation uses the same code as Length * something. Don't Repeat Yourself.
Now that the class is declared, I can get back to module code. In this case, I have some code that I want to run only if this file is executed by itself, not if it's imported as a module. So I use the following test:
if __name__ == "__main__":
Then the code in the if is executed only if this is being run directly. In this file, I have the code:
import doctest
doctest.testmod()
This goes through all the docstrings in the module and looks for lines that look like Python prompts with commands after them. The lines following are assumed to be the output of the command. If the commands output something else, the test is considered to have failed and the actual output is printed. Read the doctest module documentation for all the details.
One final note about doctests: They're useful, but they're not the most versatile or thorough tests available. For those, you'll want to read up on unittests (the unittest module).
Each python source file is a module. There are no "header" files. The basic idea is that when you import "foo" it'll load the code from "foo.py" (or a previously compiled version of it). You can then access the stuff from the foo module by saying foo.whatever.
There seem to be two ways for arranging things in Python code. Some projects use a flat layout, where all of the modules are at the top-level. Others use a hierarchy. You can import foo/bar/baz.py by importing "foo.bar.baz". The big gotcha with hierarchical layout is to have __init__.py in the appropriate directories (it can even be empty, but it should exist).
Classes are defined like this:
class MyClass(object):
def __init__(self, x):
self.x = x
def printX(self):
print self.x
To create an instance:
z = MyObject(5)
You can organize it in whatever way makes the most sense for your application. I don't exactly know what you're doing so I can't be certain what the best organization would be for you, but you can pretty much split it up as you see fit and just import what you need.
You can define classes in any file, and you can define as many classes as you would like in a script (unlike Java). There are no official header files (not like C or C++), but you can use config files to store info about connecting to a DB, whatever, and use configparser (a standard library function) to organize them.
It makes sense to keep like things in the same file, so if you have a GUI, you might have one file for the interface, and if you have a CLI, you might keep that in a file by itself. It's less important how your files are organized and more important how the source is organized into classes and functions.
This would be the place to look for that: http://docs.python.org/reference/.
First of all, compile and install pip: http://pypi.python.org/pypi/pip. It is like Ubuntu's apt-get. You run it via a Terminal by typing in pip install package-name. It has a database of packages, so you can install/uninstall stuff quite easily with it.
As for importing and "header" files, from what I can tell, if you run import foo, Python looks for foo.py in the current folder. If it's not there, it looks for eggs (folders unzipped in the Python module directory) and imports those.
As for defining classes and objects, here's a basic example:
class foo(foobar2): # I am extending a class, in this case 'foobar2'. I take no arguments.
__init__(self, the, list, of, args = True): # Instead, the arguments get passed to me. This still lets you define a 'foo()' objects with three arguments, only you let '__init__' take them.
self.var = 'foo'
def bar(self, args):
self.var = 'bar'
def foobar(self): # Even if you don't need arguments, never leave out the self argument. It's required for classes.
print self.var
foobar = foo('the', 'class', 'args') # This is how you initialize me!
Read more on this in the Python Reference, but my only tip is to never forget the self argument in class functions. It will save you a lot of debugging headaches...
Good luck!
There's no some fixed structure for Python programs, but you can take Django project as an example. Django project consists of one settings.py module, where global settings (like your example with DB connection properties) are stored and pluggable applications. Each application has it's own models.py module, which stores database models and, possibly, other domain specific objects. All the rest is up to you.
Note, that these advices are not specific to Python. In C/C++ you probably used similar structure and kept settings in XML. Just forget about headers and put settings in plain in .py file, that's all.
I'm writing some unittests for code written by someone else here at the office. Python is not my strongest language. While I've been successful with basic unit tests, mocking in python is throwing me for a loop.
What I need to do is override a call to ConfigObj and inject my own mock config/fixture into any ConfigObj call.
settings.py
from configobj import ConfigObj
config = ConfigObj('/etc/myapp/config')
utils.py
from settings import config
"""lots of stuff methods using various config values."""
What I would like to do is, in my unittests for utils.py, inject myself either for ANY call to ConfigObj or settings.py itself.
Many of the mocking libraries expect me to Mock my own classes but in the case of this app, it doesn't have any explicit classes.
Can it be done or are the python namespace restrictions too strict that I can't intervene in what a module that I'm importing imports itself?
Side note: running 2.7 so I can't do any of the tricks I've read about from 2.5.
If the tests are in a separate file from from settings.py and utils.py you can create a file mock.py
import configobj
class MockConfigObj(object):
#mock whatever you wan
configobj.ConfigObj = MockConfigObj
and then import mock before importing (from) any module that itself imports settings. This will ensure that settings.config is created with MockConfigObj. If you want a uniform global mocking, import it before any file that imports configobj.
This works because python will store configobj in sys.modules and check that before actually reading from a file on subsequent imports. in mock.py, the identifier ConfigObj is just a reference to the entry in sys.modules so that any changes that you make will be globally visible.
This strikes me as a little hacky though but it's the best that I can think of.
Python namespaces are not strict at all within the same scope. Just override the variable name containing your object (or the class itself and provided it) within the same scope you'd be expecting the original and that is good enough.
Now, whether or not what you're replacing it with behaves the same is up to you...
Couldn't you just overwrite the original function with another one?
There are no constants in Python, you can change everything, you could even do True = False.
I faced a similar situation before. Here is how I would go about addressing your problem.
Consider a test case for a function from utils.py.
import utils, unittest
class FooFunctionTests(unittest.TestCase):
def setUp(self):
utils._old_config = utils.config
utils.config = MockClass()
def tearDown(self):
utils.config = utils._old_config
del utils._old_config
def test_foo_function_returns_correct_value(self):
self.assertEqual("success!", utils.foo())
The following page is a good one on mocking and import
http://www.relaxdiego.com/2014/04/mocking-objects-in-python.html
Say you have a file named my_package1.py with the following code:
class A(object):
def init(self):
and you then import that in my_package2.py with the code
from my_package1 import A
class A(object):
def init(self):
The first line of my_package2.py creates a variable under the my_package2 namespace called A. Now you have two variables my_package1.A and my_package2.A that both point to the same class in memory. If you want the code in my_package2.py to use a mocked up class A, then you will need to mock my_package2.A not my_package1.A
That's it. If you want to document a function or a class, you put a string just after the definition. For instance:
def foo():
"""This function does nothing."""
pass
But what about a module? How can I document what a file.py does?
Add your docstring as the first statement in the module.
"""
Your module's verbose yet thorough docstring.
"""
import foo
# ...
For packages, you can add your docstring to __init__.py.
For the packages, you can document it in __init__.py.
For the modules, you can add a docstring simply in the module file.
All the information is here: http://www.python.org/dev/peps/pep-0257/
Here is an Example Google Style Python Docstrings on how module can be documented. Basically there is an information about a module, how to execute it and information about module level variables and list of ToDo items.
"""Example Google style docstrings.
This module demonstrates documentation as specified by the `Google
Python Style Guide`_. Docstrings may extend over multiple lines.
Sections are created with a section header and a colon followed by a
block of indented text.
Example:
Examples can be given using either the ``Example`` or ``Examples``
sections. Sections support any reStructuredText formatting, including
literal blocks::
$ python example_google.py
Section breaks are created by resuming unindented text. Section breaks
are also implicitly created anytime a new section starts.
Attributes:
module_level_variable1 (int): Module level variables may be documented in
either the ``Attributes`` section of the module docstring, or in an
inline docstring immediately following the variable.
Either form is acceptable, but the two should not be mixed. Choose
one convention to document module level variables and be consistent
with it.
Todo:
* For module TODOs
* You have to also use ``sphinx.ext.todo`` extension
.. _Google Python Style Guide:
http://google.github.io/styleguide/pyguide.html
"""
module_level_variable1 = 12345
def my_function():
pass
...
...
You do it the exact same way. Put a string in as the first statement in the module.
It's easy, you just add a docstring at the top of the module.
For PyPI Packages:
If you add doc strings like this in your __init__.py file as seen below
"""
Please refer to the documentation provided in the README.md,
which can be found at gorpyter's PyPI URL: https://pypi.org/project/gorpyter/
"""
# <IMPORT_DEPENDENCIES>
def setup():
"""Verify your Python and R dependencies."""
Then you will receive this in everyday usage of the help function.
help(<YOUR_PACKAGE>)
DESCRIPTION
Please refer to the documentation provided in the README.md,
which can be found at gorpyter's PyPI URL: https://pypi.org/project/gorpyter/
FUNCTIONS
setup()
Verify your Python and R dependencies.
Note, that my help DESCRIPTION is triggered by having that first docstring at the very top of the file.