I wrote my own c-module for Python and for a custom table in a documentation I need the number of parameters of the builtin-functions during runtime.
There are functions in Python 2 like inspect.getargspec or functions in Python 3 like inspect.signature which support normal Python functions, but leave builtin-functions unsupported.
There are two other community solutions so far:
Parsing the doc-strings
Parsing the original *.c file
See answer for third approach
In some cases the docstrings are outdated and/or it's hard to extract the argument count since the docstring can be any plain string. Parsing the original *.c file is a good approach as well, but you might not have access to it.
In the following this is the working solution I came up with for Python 2 and 3.
What does it do?
During runtime a list of 99 None objects gets passed to the corresponding function. One of the first checks in the internal parsing function PyArg_ParseTuple checks if the amount of parameters matches the amount of passed parameters - if not it will fail. That means we will call the function but we can also be sure it doesn't get really executed.
Technical background:
Why is it so hard to get the count of parameters of built-in functions? The problem is that the parameter list is evaluated during runtime, not compile time. A very simple example of a built-in function in C looks like this:
static PyObject* example(PyObject *self, PyObject *args)
{
int myFirstParam;
if(!PyArg_ParseTuple(args, "i", &myFirstParam))
return NULL;
...
}
Copy and Paste Solution:
import inspect
import time
import re
import types
import sys
def get_parameter_count(func):
"""Count parameter of a function.
Supports Python functions (and built-in functions).
If a function takes *args, then -1 is returned
Example:
import os
arg = get_parameter_count(os.chdir)
print(arg) # Output: 1
-- For C devs:
In CPython, some built-in functions defined in C provide
no metadata about their arguments. That's why we pass a
list with 999 None objects (randomly choosen) to it and
expect the underlying PyArg_ParseTuple fails with a
corresponding error message.
"""
# If the function is a builtin function we use our
# approach. If it's an ordinary Python function we
# fallback by using the the built-in extraction
# functions (see else case), otherwise
if isinstance(func, types.BuiltinFunctionType):
try:
arg_test = 999
s = [None] * arg_test
func(*s)
except TypeError as e:
message = str(e)
found = re.match(
r"[\w]+\(\) takes ([0-9]{1,3}) positional argument[s]* but " +
str(arg_test) + " were given", message)
if found:
return int(found.group(1))
if "takes no arguments" in message:
return 0
elif "takes at most" in message:
found = re.match(
r"[\w]+\(\) takes at most ([0-9]{1,3}).+", message)
if found:
return int(found.group(1))
elif "takes exactly" in message:
# string can contain 'takes 1' or 'takes one',
# depending on the Python version
found = re.match(
r"[\w]+\(\) takes exactly ([0-9]{1,3}|[\w]+).+", message)
if found:
return 1 if found.group(1) == "one" \
else int(found.group(1))
return -1 # *args
else:
try:
if (sys.version_info > (3, 0)):
argspec = inspect.getfullargspec(func)
else:
argspec = inspect.getargspec(func)
except:
raise TypeError("unable to determine parameter count")
return -1 if argspec.varargs else len(argspec.args)
def print_get_parameter_count(mod):
for x in dir(mod):
e = mod.__dict__.get(x)
if isinstance(e, types.BuiltinFunctionType):
print("{}.{} takes {} argument(s)".format(mod.__name__, e.__name__, get_parameter_count(e)))
import os
print_get_parameter_count(os)
Output:
os._exit takes 1 argument(s)
os.abort takes 0 argument(s)
os.access takes 2 argument(s)
os.chdir takes 1 argument(s)
os.chmod takes 2 argument(s)
os.close takes 1 argument(s)
...
Related
code being run(apart from modules and functions)
random.choice(frontright(),frontleft())
error raised:
TypeError: Random.choice() takes 2 positional arguments but 3 were given
random.choice expects a sequence to choose from. That's not what you passed it.
Functions are objects, so you can put them in a sequence and then choose from that. But if you use parens you aren't using the function itself; you are actually calling all of the functions, then passing their return values to random.choice
This code might demonstrate the difference.
import random
def frontright():
print ('frontright was called')
return 'frontright result'
def frontleft():
print ('frontleft was called')
return 'frontleft result'
my_functions = (frontright, frontleft)
fn = random.choice(my_functions)
print('calling one function')
fn()
print()
print('collection of results')
my_choice = random.choice((frontright(), frontleft()))
print('chosen result was:', my_choice)
You're actually passing 2 when it expects 1 sequence, but internally it is calling an instance method, and at that point it's really 3 instead of 2, if you include self.
You might need to do this random.choice([frontright(), frontleft()])
https://www.w3schools.com/python/ref_random_choice.asp
I have tried creating a function that tries an expression and returns zero if errors are risen.
def try_or_zero(exp):
try:
exp
return exp
except:
return 0
Which obviously doesn't work. It seems the problem is that python doesn't have any form of lazy evaluation, so the expression is evaluated before it's passed to the function and so it rises the error before it gets into the function and therefor it never passes through the try logic.
Does anyone know if this can be done in Python?
Cheers
It seems the problem is that python doesn't have any form of lazy evaluation
Err... yes it does, but possibly not in the form you expect. Function arguments ARE indeed eval'd before being passed to the function, so
try_or_zero(foo.bar())
will indeed be executed as:
param = foo.bar()
try_or_zero(param)
Now python functions are plain objects (they can be used as variables, passed around as arguments to functions etc), and they are only invoked when applying the call operator (the parens, with or without arguments) so you can pass a function to try_or_zero and let try_or_zero call the function:
def try_or_zero(func):
try:
return func()
except Exception as e:
return 0
Now you're going to object that 1/ this will not work if the function expects arguments and 2/ having to write a function just for this is a PITA - and both objections are valid. Hopefully, Python also has a shortcut to create simple anonymous functions consisting of a single (even if arbitrarily complex) expression: lambda. Also, python functions (including "lambda functions" - which are, technically, plain functions) are closure - they capture the context in which they're defined - so it's quite easy to wrap all this together:
a = 42
b = "c"
def add(x, y):
return x + y
result = try_or_zero(lambda: add(a, b))
A side note about exception handling:
First don't use a bare except, at least catch Exception (else you might prevent some exception - like SysExit- to work as expected).
Also preferably only catch the exact exceptions you expect at a given point. In your case, you may want to pass a tuple of exceptions that you want to ignore, ie:
def try_or_zero(func, *exceptions):
if not exceptions:
exceptions = (Exception,)
try:
return func()
except exceptions as e:
return 0
a = 42
b = "c"
def add(x, y):
return x + y
result = try_or_zero(lambda: add(a, b), TypeError))
which will prevent your code from masking unexpected errors.
And finally: you may also want to add support for a return value other than zero in the case of an exception (not all expressions are supposed to return an int ):
# XXX : python3 only, python2 doesn't accept
# keyword args after *args
def try_or(func, *exceptions, default=0):
if not exceptions:
exceptions = (Exception,)
try:
return func()
except exceptions as e:
return default
# adding lists is legit too,
# so here you may want an empty list as the return value
# instead
a = [1, 2, 3]
# but only to lists
b = ""
result = try_or(lambda: a + b, TypeError, default=[]))
No need to bother with exec and stuff, use the fact that python functions are objects and thus can be passed as arguments
def try_or_zero(exp):
try:
return exp()
except:
return 0
And just call try_or_zero(my_awesome_func) (without the () for your method)
Pass the argument to function in str and do the exec inside function
def try_or_zero(exp):
try:
exec(exp)
return exp
except:
return 0
so your call to function will be like below
try_or_zero('1==2')
You can achieve this by enveloping your expressions in a function.
For example:
def ErrorTest():
# the expression you want to try
raise Exception
Also your try function should look like this:
def try_catch(exp):
try :
exp() # note the paranthesis
except:
return 0
And put it inside the function
try_or_zero(ErrorTest)
OutPut: 0
You Can also do it by using the eval() function, but you will have to put your code in String.
try_or_zero(exp):
try:
eval(exp) # exp must be a string, for example 'raise ValueError'
except:
return 0
At the core, what I'm trying to do is take a number of functions that look like this undecorated validation function:
def f(k: bool):
def g(n):
# check that n is valid
return n
return g
And make them look like this decorated validation function:
#k
def f():
def g(n):
# check that n is valid
return n
return g
The idea here being that k is describing the same functionality across all of the implementing functions.
Specifically, these functions are all returning 'validation' functions for use with the voluptuous validation framework. So all the functions of type f() are returning a function that is later executed by Schema(). k is actually allow_none, which is to say a flag that determines if a None value is ok. A very simple example might be this sample use code:
x = "Some input value."
y = None
input_validator = Schema(f(allow_none=True))
x = input_validator(x) # succeeds, returning x
y = input_validator(y) # succeeds, returning None
input_validator_no_none = Schema(f(allow_none=False))
x = input_validator(x) # succeeds, returning x
y = input_validator(y) # raises an Invalid
Without changing the sample use code I am attempting to achieve the same result by changing the undecorated validation functions to decorated validation functions. To give a concrete example, changing this:
def valid_identifier(allow_none: bool=True):
min_range = Range(min=1)
validator = Any(All(int, min_range), All(Coerce(int), min_range))
return Any(validator, None) if allow_none else validator
To this:
#allow_none(default=True)
def valid_identifier():
min_range = Range(min=1)
return Any(All(int, min_range), All(Coerce(int), min_range))
The function returned from these two should be equivalent.
What I've tried to write is this, utilizing the decorator library:
from decorator import decorator
#decorator
def allow_none(default: bool=True):
def decorate_validator(wrapped_validator, allow_none: bool=default):
#wraps(wrapped_validator)
def validator_allowing_none(*args, **kwargs):
if allow_none:
return Any(None, wrapped_validator)
else:
return wrapped_validator(*args, **kwargs)
return validator_allowing_none
return decorate_validator
And I have a unittest.TestCase in order to test if this works as expected:
#allow_none()
def test_wrapped_func():
return Schema(str)
class TestAllowNone(unittest.TestCase):
def test_allow_none__success(self):
test_string = "blah"
validation_function = test_wrapped_func(allow_none=False)
self.assertEqual(test_string, validation_function(test_string))
self.assertEqual(None, validation_function(None))
But my test returns the following failure:
def validate_callable(path, data):
try:
> return schema(data)
E TypeError: test_wrapped_func() takes 0 positional arguments but 1 was given
I tried debugging this, but couldn't get the debugger to actually enter the decoration. I suspect that because of naming issues, such as raised in this (very lengthy) blog post series, that test_wrapped_func isn't getting it's argument list properly set, and so the decorator is never even executed, but it may also be something else entirely.
I tried some other variations. By removing the function parentheses from #allow_none:
#allow_none
def test_wrapped_func():
return Schema(str)
I get a different error:
> validation_function = test_wrapped_func(allow_none=False)
E TypeError: test_wrapped_func() got an unexpected keyword argument 'allow_none'
Dropping the #decorator fails with:
> validation_function = test_wrapped_func(allow_none=False)
E TypeError: decorate_validator() missing 1 required positional argument: 'wrapped_validator'
Which makes sense because #allow_none takes an argument, and so the parentheses would logically be needed. Replacing them gives the original error.
Decorators are subtle, and I'm clearly missing something here. This is similar to currying a function, but it's not quite working. What am I missing about how this should be implemented?
I think you are putting your allow_none=default argument at the wrong nesting level. It should be on the innermost function (the wrapper), rather than the decorator (the middle level).
Try something like this:
def allow_none(default=True): # this is the decorator factory
def decorator(validator): # this is the decorator
#wraps(validator)
def wrapper(*args, allow_none=default, **kwargs): # this is the wrapper
if allow_none:
return Any(None, validator)
else:
return validator(*args, **kwargs)
return wrapper
return decorator
If you don't need the default to be settable, you can get rid of the outermost layer of nesting and just make the default value a constant in the wrapper function (or omit it if your callers will always pass a value). Note that as I wrote it above, the allow_none argument to the wrapper is a keyword-only argument. If you want to pass it as a positional parameter, you can move it ahead of *args, but that requires that it be the first positional argument, which may not be desireable from an API standpoint. More sophisticated solutions are probably possible, but overkill for this answer.
I need to programmatically get the number of arguments that a function requires. With functions declared in modules this is trivial:
myfunc.func_code.co_argcount
But built-in functions don't have the func_code attribute. Is there another way to do this? Otherwise I can't use the built-ins and have to re-write them in my code.
[addition] Thanks for the responses, hope they'll be useful. I have used Pypy instead.
Take a look at the function below copied from here. This may be the best you can do. Note the comments about inspect.getargspec.
def describe_builtin(obj):
""" Describe a builtin function """
wi('+Built-in Function: %s' % obj.__name__)
# Built-in functions cannot be inspected by
# inspect.getargspec. We have to try and parse
# the __doc__ attribute of the function.
docstr = obj.__doc__
args = ''
if docstr:
items = docstr.split('\n')
if items:
func_descr = items[0]
s = func_descr.replace(obj.__name__,'')
idx1 = s.find('(')
idx2 = s.find(')',idx1)
if idx1 != -1 and idx2 != -1 and (idx2>idx1+1):
args = s[idx1+1:idx2]
wi('\t-Method Arguments:', args)
if args=='':
wi('\t-Method Arguments: None')
print
I don't believe this type of introspection is possible with built-in functions, or any C extension function for that matter.
A similar question was already asked here, and Alex's answer suggests parsing the docstring of the function to determine the number of args.
Perhaps a more robust alternative to the parsing function given by Alex, but this will still fail to give appropriate arg specs because, not all docstrings fully represent their function's signature.
A good example being dict.get where the args spec should be (k, d=None), but the function I've defined will return (k, d), because no default is given for d in the form d=None. In the docstring "f(a[, b, c])" the args b and c are defaults, but there's no real way to parse them because no value is directly specified and, in the case of dict.get, the behavior is described later on, not in the signature representation.
On the bright side though, this captures all the arguments, it's just that defaults aren't reliable.
import re
import inspect
def describe_function(function):
"""Return a function's argspec using its docstring
If usages discovered in the docstring conflict, or default
values could not be resolved, a generic argspec of *arg
and **kwargs is returned instead."""
s = function.__doc__
if s is not None:
usages = []
p = r'([\w\d]*[^\(])\( ?([^\)]*)'
for func, usage in re.findall(p, s):
if func == function.__name__:
usages.append(usage)
longest = max(usages, key=lambda s: len(s))
usages.remove(longest)
for u in usages:
if u not in longest:
# the given usages weren't subsets of a larger usage.
return inspect.ArgSpec([], 'args', 'kwargs', None)
else:
args = []
varargs = None
keywords = None
defaults = []
matchedargs = re.findall(r'( ?[^\[,\]]*) ?,? ?', longest)
for a in [a for a in matchedargs if len(a)!=0]:
if '=' in a:
name, default = a.split('=')
args.append(name)
p = re.compile(r"<\w* '(.*)'>")
m = p.match(default)
try:
if m:
d = m.groups()[0]
# if the default is a class
default = import_item(d)
else:
defaults.append(eval(default))
except:
# couldn't resolve a default value
return inspect.ArgSpec([], 'args', 'kwargs', None)
elif '**' in a:
keywords = a.replace('**', '')
elif '*' in a:
varargs = a.replace('*', '')
else:
args.append(a)
return inspect.ArgSpec(args, varargs, keywords, defaults)
# taken from traitlet.utils.importstring
def import_item(name):
"""Import and return ``bar`` given the string ``foo.bar``.
Calling ``bar = import_item("foo.bar")`` is the functional equivalent of
executing the code ``from foo import bar``.
Parameters
----------
name : string
The fully qualified name of the module/package being imported.
Returns
-------
mod : module object
The module that was imported.
"""
if not isinstance(name, string_types):
raise TypeError("import_item accepts strings, not '%s'." % type(name))
name = cast_bytes_py2(name)
parts = name.rsplit('.', 1)
if len(parts) == 2:
# called with 'foo.bar....'
package, obj = parts
module = __import__(package, fromlist=[obj])
try:
pak = getattr(module, obj)
except AttributeError:
raise ImportError('No module named %s' % obj)
return pak
else:
# called with un-dotted string
return __import__(parts[0])
It's not possible. C functions do not expose their argument signatures programmatically.
Interesting solution, ars. Hope it helps others too.
I went another way: I had heared that Pypy is python implemented mostly in itself. So I tried PyPy (JIT version) and it worked. I haven't found a "hard-coded" function yet. Couldn't find out how to install it in /usr, but it works from a folder where it's unpacked.
Is there a keyword in Matlab that is roughly equivalent to None in python?
I am trying to use it to mark an optional argument to a function. I am translating the following Python code
def f(x,y=None):
if y == None:
return g(x)
else:
return h(x,y)
into Matlab
function rtrn = f(x,y)
if y == []:
rtrn = g(x);
else
rtrn = h(x,y);
end;
end
As you can see currently I am using [] as None. Is there a better way to do this?
in your specific case. you may use nargin to determine how many input arguments here provided when calling the function.
from the MATLAB documentation:
The nargin and nargout functions
enable you to determine how many input
and output arguments a function is
called with. You can then use
conditional statements to perform
different tasks depending on the
number of arguments. For example,
function c = testarg1(a, b)
if (nargin == 1)
c = a .^ 2;
elseif (nargin == 2)
c = a + b;
end
Given a single input argument, this
function squares the input value.
Given two inputs, it adds them
together.
NaN while not equivalent, often serves the similar purpose.
nargin is definitely the easiest way of doing it. Also it is usually good practice to validate the number of input argument using nargchk:
function e = testFunc(a,b,c,d)
error( nargchk(2, 4, nargin, 'struct') );
% set default values
if nargin<4, d = 0; end
if nargin<3, c = 0; end
% ..
c = a*b + c*d;
end
... which acts as a way to ensure the correct number of arguments is passed. In this case, a minimum of two arguments are required, with a maximum of four.
If nargchk detects no error, execution resumes normally, otherwise an error is generated. For example, calling testFunc(1) generates:
Not enough input arguments.
UPDATE: A new function was introduced in R2011b narginchk, which replaces the use of the deprecated nargchk+error seen above:
narginchk(2,4);
You can use functions like: exist and isempty to check whether a variable exists and whether it is empty respectively:
if ~exist('c','var') || isempty(c)
c = 10;
end
which allows you to call your function such as: testFunc(1,2,[],4) telling it to use the default value for c but still giving a value for d
You could also use varargin to accept a variable number of arguments.
Finally a powerful way to parse and validate named inputs is to use inputParser
To see examples and other alternatives of passing arguments and setting default values, check out this post and its comments as well.
The equivalent to Python None in MATLAB is string(missing)
To test, type the following in your command window : py.type( string(missing) )
It returns <class 'NoneType'>
MATLAB to python data types documentation here
If you want to pass None into a Python function that you are calling from MATLAB, then you would pass in string(missing). This argument would show up as None in the Python function, for example, if you are detecting for None such as if arg1 == None.