I know what namespaces are. But when running
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('bar')
parser.parse_args(['XXX']) # outputs: Namespace(bar='XXX')
What kind of object is Namespace(bar='XXX')? I find this totally confusing.
Reading the argparse docs, it says "Most ArgumentParser actions add some value as an attribute of the object returned by parse_args()". Shouldn't this object then appear when running globals()? Or how can I introspect it?
Samwise's answer is very good, but let me answer the other part of the question.
Or how can I introspect it?
Being able to introspect objects is a valuable skill in any language, so let's approach this as though Namespace is a completely unknown type.
>>> obj = parser.parse_args(['XXX']) # outputs: Namespace(bar='XXX')
Your first instinct is good. See if there's a Namespace in the global scope, which there isn't.
>>> Namespace
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'Namespace' is not defined
So let's see the actual type of the thing. The Namespace(bar='XXX') printer syntax is coming from a __str__ or __repr__ method somewhere, so let's see what the type actually is.
>>> type(obj)
<class 'argparse.Namespace'>
and its module
>>> type(obj).__module__
'argparse'
Now it's a pretty safe bet that we can do from argparse import Namespace and get the type. Beyond that, we can do
>>> help(argparse.Namespace)
in the interactive interpreter to get detailed documentation on the Namespace class, all with no Internet connection necessary.
It's simply a container for the data that parse_args generates.
https://docs.python.org/3/library/argparse.html#argparse.Namespace
This class is deliberately simple, just an object subclass with a readable string representation.
Just do parser.parse_args(...).bar to get the value of your bar argument. That's all there is to that object. Per the doc, you can also convert it to a dict via vars().
The symbol Namespace doesn't appear when running globals() because you didn't import it individually. (You can access it as argparse.Namespace if you want to.) It's not necessary to touch it at all, though, because you don't need to instantiate a Namespace yourself. I've used argparse many times and until seeing this question never paid attention to the name of the object type that it returns -- it's totally unimportant to the practical applications of argparse.
Namespace is basically just a bare-bones class, on whose instances you can define attributes, with a few niceties:
A nice __repr__
Only keyword arguments can be used to instantiate it, preventing "anonymous" attributes.
A convenient method to check if an attribute exists (foo in Namespace(bar=3) evaluates to False)
Equality with other Namespace instances based on having identical attributes and attribute values. (E.g. ,Namespace(foo=3, bar=5) == Namespace(bar=5, foo=3))
Instances of Namespace are returned by parse_args:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('bar')
args = parser.parse_args(['XXX'])
assert args.bar == 'XXX'
Related
I like type hints, especially for my method parameters. In my current script one function should retrieve a parameter of type argparse._SubParsersAction. As one can see from the underscore, this is a private type by convention. I guess this is why PyCharm complains with the error message Cannot find reference '_SubParsersAction' in 'argparse.pyi' when trying to import it (although it's there).
The script runs but it feels wrong. The error message seems reasonable to me as private types are meant to be... well, private. My first question is therefore why the public method ArgumentParser.add_subparsers() returns an object of a private type in the first place.
I've looked for a public super class or interface and _SubParsersAction does indeed extend from argparse.Action, but that doesn't help me as Action does not define _SubParsersAction's add_parser() method (which I need).
So my next questions are: Can I use type hints with the argparse API? Or is it only partially possible because the API was designed long before type hints were introduced? Or does my idea of typing not fit to Python's type system?
Here is my affected code snippet. It creates an argument parser with sub arguments as described in the documentation (https://docs.python.org/dev/library/argparse.html#sub-commands).
main.py
from argparse import ArgumentParser
import sub_command_foo
import sub_command_bar
import sub_command_baz
def main():
parser = ArgumentParser()
sub_parsers = parser.add_subparsers()
sub_command_foo.add_sub_parser(sub_parsers)
sub_command_bar.add_sub_parser(sub_parsers)
sub_command_baz.add_sub_parser(sub_parsers)
args = parser.parse_args()
args.func(args)
if __name__ == '__main__':
main()
sub_command_foo.py
from argparse import _SubParsersAction, Namespace
def add_sub_parser(sub_parsers: _SubParsersAction):
arg_parser = sub_parsers.add_parser('foo')
# Add arguments...
arg_parser.set_defaults(func=run)
def run(args: Namespace):
print('foo...')
The problem lies in sub_command_foo.py. PyCharm shows the error message Cannot find reference '_SubParsersAction' in 'argparse.pyi' on the first line from argparse import _SubParsersAction, Namespace.
I don't use pycharm and haven't done much with type hints so can't help your there. But I know argparse well. The bulk of this module was written in before 2010, and it's been modified since then at a snail's pace. It's well organized in the OOP sense, but the documentation is more of glorified tutorial than a formal reference. That is, it focuses on the functions and methods users will needed, and doesn't try to formally document all classes and the methods.
I do a lot of my testing in an interactive ipython session where I can look at the objects returned by commands.
In Python the distinction between public and private classes and methods is not as formal as other languages. The '_' prefix does mark 'private' things. They aren't usually documented, but they are still accessible. Sometimes a '*' import will import all object that don't start with it, but argparse has a more explicit __all__ list.
argparser.ArgumentParser does create an object instance. But that class inherits from 2 'private' classes. The add_argument method creates an Action object, and puts it on the parser._actions list. It also returns it to the user (though usually that reference is ignored). The action is actually a subclass (all of which are 'private'). add_subparsers is just a specialized version of this add_argument.
The add_parser method creates a ArgumentParser object
It feels to me that type hinting that requires a separate import of 'private' classes is counter productive. You shouldn't be explicitly referencing those classes, even if your code produces them. Some people (companies) fear they can be changed without notification and thus break their code. Knowing how slowly argparse gets changed I wouldn't too much about that.
While trying to use introspection to navigate from strings to classes via some of the suggestions in Convert string to Python class object? I noticed that the given approaches won't work to get at a class in scope local to a function. Consider the following code:
import sys
def f():
class LocalClass:
pass
print LocalClass
print 'LocalClass' in dir(sys.modules[__name__])
which gives output
__main__.LocalClass
False
I'm a bit confused as to why LocalClass seems to belong to the main module according to the class object itself, and yet not accessible through sys.modules. Can someone give an explanation?
And is there a way to generate a class from a string, even if that class is only in non-global scope?
In the function f, LocalClass is indeed local. You can see this by trying __main__.LocalClass and seeing that AttributeError: 'module' object has no attribute 'LocalClass' is raised.
As to why the class returns __main__.LocalClass is because by default, the __repr__ function returns <cls.__module__>.<cls.__name__>.
The reason why dir isn't finding it is because it only looks at the variables defined in its scope. LocalClass is local so it won't show up if you are looking in the main module.
A way to create a class from a string can be done in many ways.
The first and easiest to understand is by using exec. Now you shouldn't just go around using exec for random things so I wouldn't reccomend using this method.
The second method is by using the type function. The help page for it returns type(name, bases, dict). This means you can create a class called LocalClass subclassed by object with the attribute foo set to "bar" by doing type("LocalClass", (object,), {"foo": "bar"}) and catching the returned class in a variable. You can make the class global by doing globals()["LocalClass"] = ...
PS: An easier (not sure if prettier) way to get the main module is by doing import __main__. This can be used in any module but I would generally advise against using this unless you know what you are doing because in general, python people don't like you doing this sort of thing.
EDIT: after looking at the linked question, you dont want to dynamically create a new class but to retrieve a variable given it's name. All the answers in the linked question will do that. I'll leave you up to deciding which one you prefer the most
EDIT2: LocalClass.__module__ is the same as __main__ because that was the module you defined the class. If you had defined it in module Foo that was imported by __main__ (and not actually ran standalone), you would find that __module__ would be "B". Even though LocalClass was defined in __main__, it won't automatically go into the global table just because it is a class - in python, as you might have already known, (almost) EVERYTHING is an object. The dir function searches for all variables defined in a scope. As you are looking in the main scope, it is nearly equivalent to be doing __dict__ or globals() but with some slight differences. Because LocalClass is local, it isn't defined in the global context. If however you did locals() whilst inside the function f, you would find that LocalClass would appear in that list
Here is some sample Python code:
import re
some_regex = re.compile(r"\s+1\s+")
result = some_regex.search(" 1 ")
dir(result)
I get back the following using Python 2.6.1:
['__copy__', '__deepcopy__', 'end', 'expand', 'group', 'groupdict', 'groups', 'span', 'start']
Yet result.re exists (from the interpreter):
>>> result.re
<_sre.SRE_Pattern object at 0x10041bc90>
How can an attribute not be listed when using the dir() function?
This page confirms the existence of the re attribute:
http://docs.python.org/library/re.html#re.MatchObject.re
Now I understand that if one tries to access an attribute which is not listed via dir(), then __getattr__ is called, but I don't see __getattr__ listed as one of the object's attributes either, so I'm left scratching my head.
Update
And here is proof of the existence of matchobject.re in the Python 2.6.1 documentation:
http://docs.python.org/release/2.6.1/library/re.html#re.MatchObject.re
You see this behavior because the class is implemented in C, and in the same way that dir() is unreliable with a custom __getattr__(), it is also unreliable when the C code defines a getattr function.
Here is a link to the Python 2.6 C code for the SRE_Match getattr function:
http://hg.python.org/cpython/file/f130ce67387d/Modules/_sre.c#l3565
Note that the methods defined in the match_methods array have Python implementations and are visible in the dir() output, but handled by an if in the match_getattr() function is not visible.
In Python 2.6, it looks like this includes the following attributes: lastindex, lastgroup, string, regs, re, pos, and endpos.
Here is a link to some of the Python 2.7 code which is slightly different. Here there is not a getattr function implemented for SRE_Match, and all methods and attributes can be found in the match_methods, match_members, and match_getset arrays, and everything is visible in dir().
http://hg.python.org/cpython/file/60a7b704de5c/Modules/_sre.c#l3612
The built-in function dir() is a convenience function and results in an approximate list of attributes. From the documentation:
Because dir() is supplied primarily as a convenience for use at an interactive prompt, it tries to supply an interesting set of names more than it tries to supply a rigorously or consistently defined set of names, and its detailed behavior may change across releases. For example, metaclass attributes are not in the result list when the argument is a class.
Note that it is impossible to always give a complete list of attributes, since classes can do in their __getattr__() and __getattribute__() methods whatever they want.
When I type help('string') in the python interpreter I get information about the string class. There,upper() is indicated as a function. Yet I can only call it as a method like "hi".upper() instead of upper("hi").
So one could assume that any method will be indicated as a function in the docstrings of the built in modules. Yet when I do help('list') , methods of the list class are indicated as methods in the docstrings!!
Why is this so? Only because the person who wrote the doctrings was inconsistent or that different people wrote it? Or do these methods(the ones called 'functions' versus the ones called 'methods' in the docstrings) actually have different properties?
When you searched for help('string'), you were looking for the docstrings of the string module. If you do help(str) or help('str') you'll get the docstrings of the str type, and here upper appears as a method!
As you can see here, the function upper from the string module is actually a function and not a method:
>>> upper('hi')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'upper' is not defined
>>> 'hi'.upper() # method from the str type
'HI'
>>> from string import upper
>>> upper('hi') # function from the string module
'HI'
You mean to do help('str'), not help('string'). str is a type, string is a module providing functions for working with strings.
You are creating an instance of that object and then calling help on that instance.
So these all work:
help(1)
help({})
help([])
help('')
help(dir)
help(help)
Help grabs the docstring for that instance, and gives it back to you.
When you create your own objects, you can put in useful docstrings or whatever you want.
There's nothing wrong with what you see.
>>> help('string')
Will show you the string module documentation. And it looks like there's an upper function inside:
>>> import string
>>> string.upper('hello')
'hello'
I'd say that this upper is the same that is called if you do:
>>> 'hello'.upper()
But I'm not sure.
Notice that a string '' is a str type not a string type. This means that you're probably looking for:
>>> help('str')
And here you'll see too the str.upper method.
This is because 'string' is a string. So is 'list'
To get a similar result for lists, try help([])
I am writing a binding system that exposes classes and functions to python in a slightly unusual way.
Normally one would create a python type and provide a list of functions that represent the methods of that type, and then allow python to use its generic tp_getattro function to select the right one.
For reasons I wont go into here, I can't do it this way, and must provide my own tp_getattro function, that selects methods from elsewhere and returns my own 'bound method' wrapper. This works fine, but means that a types methods are not listed in its dictionary (so dir(MyType()) doesn't show anything interesting).
The problem is that I cannot seem to get __add__ methods working. see the following sample:
>>> from mymod import Vec3
>>> v=Vec3()
>>> v.__add__
<Bound Method of a mymod Class object at 0xb754e080>
>>> v.__add__(v)
<mymod.Vec3 object at 0xb751d710>
>>> v+v
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'mymod.Vec3' and 'mymod.Vec3'
As you can see, Vec3 has an __add__ method which can be called, but python's + refuses to use it.
How can I get python to use it? How does the + operator actually work in python, and what method does it use to see if you can add two arbitrary objects?
Thanks.
(P.S. I am aware of other systems such as Boost.Python and SWIG which do this automatically, and I have good reason for not using them, however wonderful they may be.)
Do you have an nb_add in your type's number methods structure (pointed by field tp_as_number of your type object)?