Is there an option to print the output of help('myfun'). The behaviour I'm seeing is that output is printed to std.out and the script waits for user input (i.e. type 'q' to continue).
There must be a setting to set this to just dump docstrings.
Alternatively, if I could just dump the docstring PLUS the "def f(args):" line that would be fine too.
Searching for "python help function" is comical. :) Maybe I'm missing some nice pydoc page somewhere out there that explains it all?
To get exactly the help that's printed by help(str) into the variable strhelp:
import pydoc
strhelp = pydoc.render_doc(str, "Help on %s")
Of course you can then easily print it without paging, etc.
You've already seen reference to the docstring, the magic __doc__ variable which holds the body of the help:
def foo(a,b,c):
''' DOES NOTHING!!!! '''
pass
print foo.__doc__ # DOES NOTHING!!!!
To get the name of a function, you just use __name__:
def foo(a,b,c): pass
print foo.__name__ # foo
The way to get the signature of a function which is not built in you can use the func_code property and from that you can read its co_varnames:
def foo(a,b,c): pass
print foo.func_code.co_varnames # ('a', 'b', 'c')
I've not found out how to do the same for built in functions.
If you want to access the raw docstring from code:
myvar = obj.__doc__
print(obj.__doc__)
The help function does some additional processing, the accepted answer shows how to replicate this with pydoc.render_doc().
>>> x = 2
>>> x.__doc__
'int(x[, base]) -> integer\n\nConvert a string or number to an integer, if possi
ble. A floating point\nargument will be truncated towards zero (this does not i
nclude a string\nrepresentation of a floating point number!) When converting a
string, use\nthe optional base. It is an error to supply a base when converting
a\nnon-string. If the argument is outside the integer range a long object\nwill
be returned instead.'
Is that what you needed?
edit - you can print(x.__doc__) and concerning the function signature, you can build it using the inspect module.
>>> inspect.formatargspec(inspect.getargspec(os.path.join))
'((a,), p, None, None)'
>>> help(os.path.join)
Help on function join in module ntpath:
join(a, *p)
Join two or more pathname components, inserting "\" as needed
Related
Note: This question is for informational purposes only. I am interested to see how deep into Python's internals it is possible to go with this.
Not very long ago, a discussion began inside a certain question regarding whether the strings passed to print statements could be modified after/during the call to print has been made. For example, consider the function:
def print_something():
print('This cat was scared.')
Now, when print is run, then the output to the terminal should display:
This dog was scared.
Notice the word "cat" has been replaced by the word "dog". Something somewhere somehow was able to modify those internal buffers to change what was printed. Assume this is done without the original code author's explicit permission (hence, hacking/hijacking).
This comment from the wise #abarnert, in particular, got me thinking:
There are a couple of ways to do that, but they're all very ugly, and
should never be done. The least ugly way is to probably replace the
code object inside the function with one with a different co_consts
list. Next is probably reaching into the C API to access the str's
internal buffer. [...]
So, it looks like this is actually possible.
Here's my naive way of approaching this problem:
>>> import inspect
>>> exec(inspect.getsource(print_something).replace('cat', 'dog'))
>>> print_something()
This dog was scared.
Of course, exec is bad, but that doesn't really answer the question, because it does not actually modify anything during when/after print is called.
How would it be done as #abarnert has explained it?
First, there's actually a much less hacky way. All we want to do is change what print prints, right?
_print = print
def print(*args, **kw):
args = (arg.replace('cat', 'dog') if isinstance(arg, str) else arg
for arg in args)
_print(*args, **kw)
Or, similarly, you can monkeypatch sys.stdout instead of print.
Also, nothing wrong with the exec … getsource … idea. Well, of course there's plenty wrong with it, but less than what follows here…
But if you do want to modify the function object's code constants, we can do that.
If you really want to play around with code objects for real, you should use a library like bytecode (when it's finished) or byteplay (until then, or for older Python versions) instead of doing it manually. Even for something this trivial, the CodeType initializer is a pain; if you actually need to do stuff like fixing up lnotab, only a lunatic would do that manually.
Also, it goes without saying that not all Python implementations use CPython-style code objects. This code will work in CPython 3.7, and probably all versions back to at least 2.2 with a few minor changes (and not the code-hacking stuff, but things like generator expressions), but it won't work with any version of IronPython.
import types
def print_function():
print ("This cat was scared.")
def main():
# A function object is a wrapper around a code object, with
# a bit of extra stuff like default values and closure cells.
# See inspect module docs for more details.
co = print_function.__code__
# A code object is a wrapper around a string of bytecode, with a
# whole bunch of extra stuff, including a list of constants used
# by that bytecode. Again see inspect module docs. Anyway, inside
# the bytecode for string (which you can read by typing
# dis.dis(string) in your REPL), there's going to be an
# instruction like LOAD_CONST 1 to load the string literal onto
# the stack to pass to the print function, and that works by just
# reading co.co_consts[1]. So, that's what we want to change.
consts = tuple(c.replace("cat", "dog") if isinstance(c, str) else c
for c in co.co_consts)
# Unfortunately, code objects are immutable, so we have to create
# a new one, copying over everything except for co_consts, which
# we'll replace. And the initializer has a zillion parameters.
# Try help(types.CodeType) at the REPL to see the whole list.
co = types.CodeType(
co.co_argcount, co.co_kwonlyargcount, co.co_nlocals,
co.co_stacksize, co.co_flags, co.co_code,
consts, co.co_names, co.co_varnames, co.co_filename,
co.co_name, co.co_firstlineno, co.co_lnotab,
co.co_freevars, co.co_cellvars)
print_function.__code__ = co
print_function()
main()
What could go wrong with hacking up code objects? Mostly just segfaults, RuntimeErrors that eat up the whole stack, more normal RuntimeErrors that can be handled, or garbage values that will probably just raise a TypeError or AttributeError when you try to use them. For examples, try creating a code object with just a RETURN_VALUE with nothing on the stack (bytecode b'S\0' for 3.6+, b'S' before), or with an empty tuple for co_consts when there's a LOAD_CONST 0 in the bytecode, or with varnames decremented by 1 so the highest LOAD_FAST actually loads a freevar/cellvar cell. For some real fun, if you get the lnotab wrong enough, your code will only segfault when run in the debugger.
Using bytecode or byteplay won't protect you from all of those problems, but they do have some basic sanity checks, and nice helpers that let you do things like insert a chunk of code and let it worry about updating all offsets and labels so you can't get it wrong, and so on. (Plus, they keep you from having to type in that ridiculous 6-line constructor, and having to debug the silly typos that come from doing so.)
Now on to #2.
I mentioned that code objects are immutable. And of course the consts are a tuple, so we can't change that directly. And the thing in the const tuple is a string, which we also can't change directly. That's why I had to build a new string to build a new tuple to build a new code object.
But what if you could change a string directly?
Well, deep enough under the covers, everything is just a pointer to some C data, right? If you're using CPython, there's a C API to access the objects, and you can use ctypes to access that API from within Python itself, which is such a terrible idea that they put a pythonapi right there in the stdlib's ctypes module. :) The most important trick you need to know is that id(x) is the actual pointer to x in memory (as an int).
Unfortunately, the C API for strings won't let us safely get at the internal storage of an already-frozen string. So screw safely, let's just read the header files and find that storage ourselves.
If you're using CPython 3.4 - 3.7 (it's different for older versions, and who knows for the future), a string literal from a module that's made of pure ASCII is going to be stored using the compact ASCII format, which means the struct ends early and the buffer of ASCII bytes follows immediately in memory. This will break (as in probably segfault) if you put a non-ASCII character in the string, or certain kinds of non-literal strings, but you can read up on the other 4 ways to access the buffer for different kinds of strings.
To make things slightly easier, I'm using the superhackyinternals project off my GitHub. (It's intentionally not pip-installable because you really shouldn't be using this except to experiment with your local build of the interpreter and the like.)
import ctypes
import internals # https://github.com/abarnert/superhackyinternals/blob/master/internals.py
def print_function():
print ("This cat was scared.")
def main():
for c in print_function.__code__.co_consts:
if isinstance(c, str):
idx = c.find('cat')
if idx != -1:
# Too much to explain here; just guess and learn to
# love the segfaults...
p = internals.PyUnicodeObject.from_address(id(c))
assert p.compact and p.ascii
addr = id(c) + internals.PyUnicodeObject.utf8_length.offset
buf = (ctypes.c_int8 * 3).from_address(addr + idx)
buf[:3] = b'dog'
print_function()
main()
If you want to play with this stuff, int is a whole lot simpler under the covers than str. And it's a lot easier to guess what you can break by changing the value of 2 to 1, right? Actually, forget imagining, let's just do it (using the types from superhackyinternals again):
>>> n = 2
>>> pn = PyLongObject.from_address(id(n))
>>> pn.ob_digit[0]
2
>>> pn.ob_digit[0] = 1
>>> 2
1
>>> n * 3
3
>>> i = 10
>>> while i < 40:
... i *= 2
... print(i)
10
10
10
… pretend that code box has an infinite-length scrollbar.
I tried the same thing in IPython, and the first time I tried to evaluate 2 at the prompt, it went into some kind of uninterruptable infinite loop. Presumably it's using the number 2 for something in its REPL loop, while the stock interpreter isn't?
Monkey-patch print
print is a builtin function so it will use the print function defined in the builtins module (or __builtin__ in Python 2). So whenever you want to modify or change the behavior of a builtin function you can simply reassign the name in that module.
This process is called monkey-patching.
# Store the real print function in another variable otherwise
# it will be inaccessible after being modified.
_print = print
# Actual implementation of the new print
def custom_print(*args, **options):
_print('custom print called')
_print(*args, **options)
# Change the print function globally
import builtins
builtins.print = custom_print
After that every print call will go through custom_print, even if the print is in an external module.
However you don't really want to print additional text, you want to change the text that is printed. One way to go about that is to replace it in the string that would be printed:
_print = print
def custom_print(*args, **options):
# Get the desired seperator or the default whitspace
sep = options.pop('sep', ' ')
# Create the final string
printed_string = sep.join(args)
# Modify the final string
printed_string = printed_string.replace('cat', 'dog')
# Call the default print function
_print(printed_string, **options)
import builtins
builtins.print = custom_print
And indeed if you run:
>>> def print_something():
... print('This cat was scared.')
>>> print_something()
This dog was scared.
Or if you write that to a file:
test_file.py
def print_something():
print('This cat was scared.')
print_something()
and import it:
>>> import test_file
This dog was scared.
>>> test_file.print_something()
This dog was scared.
So it really works as intended.
However, in case you only temporarily want to monkey-patch print you could wrap this in a context-manager:
import builtins
class ChangePrint(object):
def __init__(self):
self.old_print = print
def __enter__(self):
def custom_print(*args, **options):
# Get the desired seperator or the default whitspace
sep = options.pop('sep', ' ')
# Create the final string
printed_string = sep.join(args)
# Modify the final string
printed_string = printed_string.replace('cat', 'dog')
# Call the default print function
self.old_print(printed_string, **options)
builtins.print = custom_print
def __exit__(self, *args, **kwargs):
builtins.print = self.old_print
So when you run that it depends on the context what is printed:
>>> with ChangePrint() as x:
... test_file.print_something()
...
This dog was scared.
>>> test_file.print_something()
This cat was scared.
So that's how you could "hack" print by monkey-patching.
Modify the target instead of the print
If you look at the signature of print you'll notice a file argument which is sys.stdout by default. Note that this is a dynamic default argument (it really looks up sys.stdout every time you call print) and not like normal default arguments in Python. So if you change sys.stdout print will actually print to the different target even more convenient that Python also provides a redirect_stdout function (from Python 3.4 on, but it's easy to create an equivalent function for earlier Python versions).
The downside is that it won't work for print statements that don't print to sys.stdout and that creating your own stdout isn't really straightforward.
import io
import sys
class CustomStdout(object):
def __init__(self, *args, **kwargs):
self.current_stdout = sys.stdout
def write(self, string):
self.current_stdout.write(string.replace('cat', 'dog'))
However this also works:
>>> import contextlib
>>> with contextlib.redirect_stdout(CustomStdout()):
... test_file.print_something()
...
This dog was scared.
>>> test_file.print_something()
This cat was scared.
Summary
Some of these points have already be mentioned by #abarnet but I wanted to explore these options in more detail. Especially how to modify it across modules (using builtins/__builtin__) and how to make that change only temporary (using contextmanagers).
A simple way to capture all output from a print function and then process it, is to change the output stream to something else, e.g. a file.
I'll use a PHP naming conventions (ob_start, ob_get_contents,...)
from functools import partial
output_buffer = None
print_orig = print
def ob_start(fname="print.txt"):
global print
global output_buffer
print = partial(print_orig, file=output_buffer)
output_buffer = open(fname, 'w')
def ob_end():
global output_buffer
close(output_buffer)
print = print_orig
def ob_get_contents(fname="print.txt"):
return open(fname, 'r').read()
Usage:
print ("Hi John")
ob_start()
print ("Hi John")
ob_end()
print (ob_get_contents().replace("Hi", "Bye"))
Would print
Hi John
Bye John
Let's combine this with frame introspection!
import sys
_print = print
def print(*args, **kw):
frame = sys._getframe(1)
_print(frame.f_code.co_name)
_print(*args, **kw)
def greetly(name, greeting = "Hi")
print(f"{greeting}, {name}!")
class Greeter:
def __init__(self, greeting = "Hi"):
self.greeting = greeting
def greet(self, name):
print(f"{self.greeting}, {name}!")
You'll find this trick prefaces every greeting with the calling function or method. This might be very useful for logging or debugging; especially as it lets you "hijack" print statements in third party code.
I am trying to make a script in Python, that when executed, asks the user for a function name and prints the function .__doc__
For example:
>>> print abs.__doc__
abs(number) -> number
Return the absolute value of the argument.
The problem is, it doesn't work with raw_input. The following is my code and what happens when it gets executed.
Code:
f = raw_input("Your function: ")
print f.__doc__
Execution:
Your function: abs
str(object='') -> string
Return a nice string representation of the object.
If the argument is a string, the return value is the same object.
What am I doing wrong?
Well you ask to print the __doc__ of f and f is in this case something like 'abs'. So you call 'abs'.__doc__ which is the __doc__ of a string.
Now you can however query for a builtin function with:
func = getattr(__builtins__,f)
print func.__doc__
this will however only work for builtin functions. You can also look for globals() or locals() which are dictionaries storing the global and local variables respectively.
As others have said, your problem is trying to use the string reply from raw_input() as a function object. You could get the function object by calling getattr(), but you need to known which module it is in. OK, so you could go through the global namespace looking for it, but there is a much simpler solution, just use pydoc:
f = raw_input("Your function: ")
help(f)
In this case, abs is a method that you're querying the __doc__ on. The raw_input is converting the input value to a string, so what you're really executing the __doc__ on is a string.
You get the same results if you do this:
z = ''
print z.__doc__
I know that the parameters can be any object but for the documentation it is quite important to specify what you would expect.
First is how to specify a parameter types like these below?
str (or use String or string?)
int
list
dict
function()
tuple
object instance of class MyClass
Second, how to specify params that can be of multiple types like a function that can handle a single parameter than can be int or str?
Please use the below example to demonstrate the syntax needed for documenting this with your proposed solution. Mind that it is desired to be able to hyperlink reference to the "Image" class from inside the documentation.
def myMethod(self, name, image):
"""
Does something ...
name String: name of the image
image Image: instance of Image Class or a string indicating the filename.
Return True if operation succeeded or False.
"""
return True
Note, you are welcome to suggest the usage of any documentation tool (sphinx, oxygen, ...) as long it is able to deal with the requirements.
Update:
It seams that there is some kind of support for documenting parameter types in doxygen in. general. The code below works but adds an annoying $ to the param name (because it was initially made for php).
#param str $arg description
#param str|int $arg description
There is a better way. We use
def my_method(x, y):
"""
my_method description
#type x: int
#param x: An integer
#type y: int|string
#param y: An integer or string
#rtype: string
#return: Returns a sentence with your variables in it
"""
return "Hello World! %s, %s" % (x,y)
That's it. In the PyCharm IDE this helps a lot. It works like a charm ;-)
You need to add an exclamation mark at the start of the Python docstring for Doxygen to parse it correctly.
def myMethod(self, name, image):
"""!
Does something ...
#param name String: name of the image
#param image Image: instance of Image Class or a string indicating the filename.
#return Return True if operation succeeded or False.
"""
return True
If using Python 3, you can use the function annotations described in PEP 3107.
def compile(
source: "something compilable",
filename: "where the compilable thing comes from",
mode: "is this a single statement or a suite?"):
See also function definitions.
Figured I'd post this little tidbit here since IDEA showed me this was possible, and I was never told nor read about this.
>>> def test( arg: bool = False ) -> None: print( arg )
>>> test(10)
10
When you type test(, IDLE's doc-tip appears with (arg: bool=False) -> None Which was something I thought only Visual Studio did.
It's not exactly doxygen material, but it's good for documenting parameter-types for those using your code.
Yup, #docu is right - this is the (IMHO best) way to combine both documentation schemes more or less seamlessly. If, on the other hand, you also want to do something like putting text on the doxygen-generated index page, you would add
##
# #mainpage (Sub)Heading for the doxygen-generated index page
# Text that goes right onto the doxygen-generated index page
somewhere at the beginning of your Python code.
In other words, where doxygen does not expect Python comments, use ## to alert it that there are tags for it. Where it expects Python comments (e.g. at the beginning of functions or classes), use """!.
Doxygen is great for C++, but if you are working with mostly python code you should give sphinx a try. If you choose sphinx then all you need to do is follow pep8.
I want to have a function in a different module, that when called, has access to all variables that its caller has access to, and functions just as if its body had been pasted into the caller rather than having its own context, basically like a C Macro instead of a normal function. I know I can pass locals() into the function and then it can access the local variables as a dict, but I want to be able to access them normally (eg x.y, not x["y"] and I want all names the caller has access to not just the locals, as well as things that were 'imported' into the caller's file but not into the module that contains the function.
Is this possible to pull off?
Edit 2 Here's the simplest possible example I can come up with of what I'm really trying to do:
def getObj(expression)
ofs = expression.rfind(".")
obj = eval(expression[:ofs])
print "The part of the expression Left of the period is of type ", type(obj),
Problem is that 'expression' requires the imports and local variables of the caller in order to eval without error.In reality theres a lot more than just an eval, so I'm trying to avoid the solution of just passing locals() in and through to the eval() since that won't fix my general case problem.
And another, even uglier way to do it -- please don't do this, even if it's possible --
import sys
def insp():
l = sys._getframe(1).f_locals
expression = l["expression"]
ofs = expression.rfind(".")
expofs = expression[:ofs]
obj = eval(expofs, globals(), l)
print "The part of the expression %r Left of the period (%r) is of type %r" % (expression, expofs, type(obj)),
def foo():
derp = 5
expression = "derp.durr"
insp()
foo()
outputs
The part of the expression 'derp.durr' Left of the period ('derp') is of type (type 'int')
I don't presume this is the answer that you wanted to hear, but trying to access local variables from a caller module's scope is not a good idea. If you normally program in PHP or C, you might be used to this sort of thing?
If you still want to do this, you might consider creating a class and passing an instance of that class in place of locals():
#other_module.py
def some_func(lcls):
print(lcls.x)
Then,
>>> import other_module
>>>
>>>
>>> x = 'Hello World'
>>>
>>> class MyLocals(object):
... def __init__(self, lcls):
... self.lcls = lcls
... def __getattr__(self, name):
... return self.lcls[name]
...
>>> # Call your function with an instance of this instead.
>>> other_module.some_func(MyLocals(locals()))
'Hello World'
Give it a whirl.
Is this possible to pull off?
Yes (sort of, in a very roundabout way) which I would strongly advise against it in general (more on that later).
Consider:
myfile.py
def func_in_caller():
print "in caller"
import otherfile
globals()["imported_func"] = otherfile.remote_func
imported_func(123, globals())
otherfile.py
def remote_func(x1, extra):
for k,v in extra.iteritems():
globals()[k] = v
print x1
func_in_caller()
This yields (as expected):
123
in caller
What we're doing here is trickery: we just copy every item into another namespace in order to make this work. This can (and will) break very easily and/or lead to hard to find bugs.
There's almost certainly a better way of solving your problem / structuring your code (we need more information in general on what you're trying to achieve).
From The Zen of Python:
2) Explicit is better than implicit.
In other words, pass in the parameter and don't try to get really fancy just because you think it would be easier for you. Writing code is not just about you.
Is there a way to pass a list as a function argument to eval() Or do I have to convert it to a string and then parse it as a list in the function?
My simple example looks like:
eval("func1(\'" + fArgs + "\')")
I'm just not sure if there is a better way of taking fArgs as a list instead of a string
Note:
The list is provided from a JSON response
EDIT: Ok here's a bit more of my class so there's a better understanding of how I'm using eval
def test(arg):
print arg
#Add all allowed functions to this list to be mapped to a dictionary
safe_list = ['test']
safe_dict = dict([ (k, locals().get(k, None)) for k in safe_list ])
class Validate:
def __init__(self, Value, fName, fArgs):
eval(fName + "(\'" + fArgs + "\')", {"__builtins__":None},safe_dict)
I may be wrong in thinking this, but to my understanding this is a safe use of eval because the only functions that can be called are the ones that are listed in the safe_list dictionary. The function to be run and the arguments for that function are being extracted out of a JSON object. The arguments are to be structured as a list, Will joining the list together with ", " be interpreted as actual arguments or just a single argument?
If you're using Python 2.6.x, then you should be able to use the json module (see py doc 19.2). If not, then there is python-json available through the python package index. Both of these packages will provide a reader for parsing JSON data into an appropriate Python data type.
For your second problem of calling a function determined by a message, you can do the following:
def foo():
print 'I am foo!'
def bar():
pass
def baz():
pass
funcs = {'func_a':foo, 'func_b':bar, 'func_c':baz}
funcs['func_a']()
This approach can be a bit more secure than eval because it prevents 'unsafe' python library functions from being injected into the JSON. However, you still need to be cautious that the data supplied to your functions can't be manipulated to cause problems.
Specifying parameters the following way works:
root#parrot$ more test.py
def func1(*args):
for i in args:
print i
l = [1,'a',9.1]
func1(*l)
root#parrot$ python test.py
1
a
9.1
so, no direct need for eval(), unless I'm misunderstanding something.
Using a library to parse JSON input may be a better approach than eval, something like:
import json
func1(json.loads(fArgs))
Assert-ing that user input is correct would be a good idea, too.
The others have a good point, that you shouldn't be using eval. But, if you must:
eval("func1(%s)" % ", ".join(fArgs))
will call the function with all the arguments in the list. This:
eval("func1([%s])" % ", ".join(fArgs))
will call it with the list of arguments in just one argument. Maybe you even want this?
eval("func1([%s])" % ", ".join(map(eval, fArgs)))
which would eval the arguments as well?