This is more of a curiosity question than anything else. I'm new with Python and playing around with it. I've just looked at the base64 module. What if instead of doing:
import base64
string = 'Foo Bar'
encoded = base664.b64encode
I wanted to do something like:
>>> class b64string():
>>> <something>
>>>
>>> string = b64string('Foo Bar')
>>> string
'Foo Bar'
>>> string.encode64()
'Rm9vIEJhcg=='
>>> string
'Rm9vIEJhcg=='
>>> string.assign('QmFyIEZvbw==')
>>> string
'QmFyIEZvbw=='
>>> string.b64decode()
'Bar Foo'
>>> string
'Bar Foo'
Is there a simple, pythonic way to create that class?
I've begun with this:
>>> class b64string(base64):
... def __init__(self, v):
... self.value=v
And already I get:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
module.__init__() takes at most 2 arguments (3 given)
And don't get me started on (just to see what would happen):
>>> class b64string(str, base64): pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
I know how to do it manually by listing all of the attributes of base64 in a new class and calling them with the stored value as argument. But is there a neat, pythonic way to do this? Is it a bad idea to do it? The idea would be, if needed, to do it with many such modules and have "super strings" that would have as modules all the things I would need to do with them. Is that bad? Is it un-pythonic? If it is pythonic, how is it done?
I don't think creating such complex string-like classes is a good idea, but if you really want to, here's a simple snippet that runs your examples.
First, we define a class that's a generic string-wrapper. Its core is a __getattr__ function that forwards every method call to a given self.module, adding self.string as the first parameter and remembering the result on self.string.
import base64
class ModuledString(object):
def __init__(self, string):
self.string = string
def __getattr__(self, attrname):
def func(*args, **kwargs):
result = getattr(self.module, attrname)(self.string, *args, **kwargs)
self.string = result
return result
return func
def __str__(self):
return str(self.string)
Creating a string-wrapper with base64 capabilities is then easy:
class B64String(ModuledString):
module = base64
if __name__ == '__main__':
string = B64String('Foo Bar')
print string
# 'Foo Bar'
print string.b64encode()
# 'Rm9vIEJhcg=='
print string
# 'Rm9vIEJhcg=='
string.string = 'QmFyIEZvbw=='
print string
# 'QmFyIEZvbw=='
print string.b64decode()
# 'Bar Foo'
Note that the above examples work only because b64encode and b64decode take a string as the first argument and return a string as the result (there is no validation in my __getattr__ function). A random function from some random module would probably raise some kind of exception. So, after all, it would be better to restrict the usage to a predefined set of functions from a given module, but it should be easy now.
I repeat, I don't recommend using such code in any serious project, only for fun.
Related
I would like to store a bunch of variables under a Python namespace without creating a separate module. I notice that the result of ArgumentParser's parse_args() is a argparse.Namespace object. You can access the arguments through dot-syntax.
from argparse import ArgumentParser
parser = ArgumentParser()
# some arg definitions here...
args = parser.parse_args() # returns a `argparse.Namespace` object
How can I create the equivalent of an argparse.Namespace? I know I can do something similar with a dict but I would like to use dot-syntax. Is there any built-in class that just lets you assign arbitrary attributes?
Starting with python3.3 you can use types.SimpleNamespace.
However an alternative is simply:
class Namespace(object):
pass
namespaceA = Namespace()
namespaceA.x = 1
The full code for SimpleNamespace isn't much longer.
Note that you cannot simply use an object instance:
>>> o = object()
>>> o.x = 1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'object' object has no attribute 'x'
This is because instances of object do not have a __dict__ attribute:
>>> vars(object())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: vars() argument must have __dict__ attribute
Which means you cannot set the attributes of an instance of object.
Any object subclass that does not have the __slots__ attribute set does have the __dict__ which is used (by default) to store/retrieve attributes:
>>> class Namespace(object):
... pass
...
>>> a = Namespace()
>>> a.x = 1 # same as as.__dict__['a'] = 1
>>> a.__dict__
{'x': 1}
For further information about attribute setting/lookup you should learn about descriptors.
A class can be used as a namespace, where the variables are class members:
class Namespace1:
foo = 'a'
bar = 5
To prevent callers from trying to instantiate, you can use a baseclass like:
class objectless(object):
def __new__(cls, *args, **kwargs):
raise RuntimeError('%s should not be instantiated' % cls)
And use it like:
class Namespace1(objectless):
...
It sounds like you want a python class. See the docs.
Depending on what you want exactly, you can define a bunch of variables as attributes of a class (either a variable of an instance or of the class itself) and access them that way.
If you want "the equivalent of an argparse.Namespace", use argparse.Namespace:
from argparse import Namespace
ns = Namespace(a=1)
print ns.a
If I'm understanding correctly, you want to dynamically add attributes to it. For example, a class parses command-line flags you access them directly like args.verbose, right? If so, you may be thinking of setattr() that lets you add arbitrary attributes.
class Foo(object):
pass
foo = Foo()
setattr(foo, 'ack', 'bar')
print(foo.ack) # prints 'bar'
I've been trying to get some dynamically created types (i.e. ones created by calling 3-arg type()) to pickle and unpickle nicely. I've been using this module switching trick to hide the details from users of the module and give clean semantics.
I've learned several things already:
The type must be findable with getattr on the module itself
The type must be consistent with what getattr finds, that is to say if we call pickle.dumps(o) then it must be true that type(o) == getattr(module, 'name of type')
Where I'm stuck though is that there still seems to be something odd going on - it seems to be calling __getstate__ on something unexpected.
Here's the simplest setup I've got that reproduces the issue, testing with Python 3.5, but I'd like to target back to 3.3 if possible:
# module.py
import sys
import functools
def dump(self):
return b'Some data' # Dummy for testing
def undump(self, data):
print('Undump: %r' % data) # Do nothing for testing
# Cheaty demo way to make this consistent
#functools.lru_cache(maxsize=None)
def make_type(name):
return type(name, (), {
'__getstate__': dump,
'__setstate__': undump,
})
class Magic(object):
def __init__(self, path):
self.path = path
def __getattr__(self, name):
print('Getting thing: %s (from: %s)' % (name, self.path))
# for simple testing all calls to make_type must end in last x.y.z.last
if name != 'last':
if self.path:
return Magic(self.path + '.' + name)
else:
return Magic(name)
return make_type(self.path + '.' + name)
# Make the switch
sys.modules[__name__] = Magic('')
And then a quick way to exercise that:
import module
import pickle
f=module.foo.bar.woof.last()
print(f.__getstate__()) # See, *this* works
print('Pickle starts here')
print(pickle.dumps(f))
Which then gives:
Getting thing: foo (from: )
Getting thing: bar (from: foo)
Getting thing: woof (from: foo.bar)
Getting thing: last (from: foo.bar.woof)
b'Some data'
Pickle starts here
Getting thing: __spec__ (from: )
Getting thing: _initializing (from: __spec__)
Getting thing: foo (from: )
Getting thing: bar (from: foo)
Getting thing: woof (from: foo.bar)
Getting thing: last (from: foo.bar.woof)
Getting thing: __getstate__ (from: foo.bar.woof)
Traceback (most recent call last):
File "test.py", line 7, in <module>
print(pickle.dumps(f))
TypeError: 'Magic' object is not callable
I wasn't expecting to see anything looking up __getstate__ on module.foo.bar.woof, but even if we force that lookup to fail by adding:
if name == '__getstate__': raise AttributeError()
into our __getattr__ it still fails with:
Traceback (most recent call last):
File "test.py", line 7, in <module>
print(pickle.dumps(f))
_pickle.PicklingError: Can't pickle <class 'module.Magic'>: it's not the same object as module.Magic
What gives? Am I missing something with __spec__? The docs for __spec__ pretty much just stress setting it appropriately, but don't seem to actually explain much.
More importantly the bigger question is how am I supposed to go about making types I programatically generated via a pseudo module's __getattr__ implementation pickle properly?
(And obviously once I've managed to get pickle.dumps to produce something I expect pickle.loads to call undump with the same thing)
To pickle f, pickle needs to pickle f's class, module.foo.bar.woof.last.
The docs don't claim support for pickling arbitrary classes. They claim the following:
The following types can be pickled:
...
classes that are defined at the top level of a module
module.foo.bar.woof.last isn't defined at the top level of a module, even a pretend module like module. In this not-officially-supported case, the pickle logic ends up trying to pickle module.foo.bar.woof, either here:
elif parent is not module:
self.save_reduce(getattr, (parent, lastname))
or here
else if (parent != module) {
PickleState *st = _Pickle_GetGlobalState();
PyObject *reduce_value = Py_BuildValue("(O(OO))",
st->getattr, parent, lastname);
status = save_reduce(self, reduce_value, NULL);
module.foo.bar.woof can't be pickled for multiple reasons. It returns a non-callable Magic instance for all unsupported method lookups, like __getstate__, which is where your first error comes from. The module-switching thing prevents finding the Magic class to pickle it, which is where your second error comes from. There are probably more incompatibilities.
As it seems, and is already proven that making the class callable is just a drifting out another wrong direction, thankfully to this hack, I could find a getaround to make the class reiterable by its TYPE. following the context of the error <class 'module.Magic'>: it's not the same object as module.Magic the pickler doesn't iterate through the same call that renders a different type from the other one, this is a major common problem with pickling self instanciating classes, for this instance, an object by its class, there for the solution is patching the class with its type #mock.patch('module.Magic', type(module.Magic)) this is a short answer for a something.
Main.py
import module
import pickle
import mock
f=module1.foo.bar.woof.last
print(f().__getstate__()) # See, *this* works
print('Pickle starts here')
#mock.patch('module1.Magic', type(module1.Magic))
def pickleit():
return pickle.dumps(f())
print(pickleit())
Magic class
class Magic(object):
def __init__(self, value):
self.path = value
__class__: lambda x:x
def __getstate__(self):
print ("Shoot me! i'm at " + self.path )
return dump(self)
def __setstate__(self,value):
print ('something will never occur')
return undump(self,value)
def __spec__(self):
print ("Wrong side of the planet ")
def _initializing(self):
print ("Even farther lost ")
def __getattr__(self, name):
print('Getting thing: %s (from: %s)' % (name, self.path))
# for simple testing all calls to make_type must end in last x.y.z.last
if name != 'last':
if self.path:
return Magic(self.path + '.' + name)
else:
return Magic(name)
print('terminal stage' )
return make_type(self.path + '.' + name)
Even assuming this is not more of striking the ball by the edge of the bat, I could see the content dumped into my console.
A bit of background
I'm writing a python module for my own use, and I'm using Python's logging module. There are handlers and formatters and even a pair of functions I create that (for the most part) won't be used anywhere else. However, I still want to be able to access and modify these variables elsewhere (for instance, other closely-coupled modules or scripts)
A simple namespace
What I'm currently doing is using a class definition to group all of my variables together, like this:
class _Logging:
'''A little namespace for our logging facilities. Don't try to instantiate
it: all it does is group together some logging objects and keep them out of
the global namespace'''
global logger
def __init__(self):
raise TypeError("that's not how this works...")
def gz_log_rotator(source, dest):
'''accept a source filename and a destination filename. copy source to
dest and add gzip compression. for use with
logging.handlers.RotatingFileHandler.rotator.'''
with gzip.open(dest, 'wb', 1) as ofile, open(source, 'rb') as ifile:
ofile.write(ifile.read())
os.remove(source)
def gz_log_namer(name):
'''accept a filename, and return it with ".gz" appended. for use with
logging.handlers.RotatingFileHandler.namer.'''
return name + ".gz"
fmtr = logging.Formatter(
'[%(asctime)s:%(name)s:%(thread)05d:%(levelname)-8s] %(message)s')
gz_rotfile_loghandler = logging.handlers.RotatingFileHandler(
'%s.log' % __name__, mode='a', maxBytes=(1024**2 * 20), backupCount=3)
gz_rotfile_loghandler.setLevel(5)
gz_rotfile_loghandler.setFormatter(fmtr)
gz_rotfile_loghandler.rotator = gz_log_rotator
gz_rotfile_loghandler.namer = gz_log_namer
simplefile_loghandler = logging.FileHandler(
'%s.simple.log' % __name__, mode='w')
simplefile_loghandler.setLevel(15)
simplefile_loghandler.setFormatter(fmtr)
stream_loghandler = logging.StreamHandler()
stream_loghandler.setLevel(25)
stream_loghandler.setFormatter(fmtr)
logger = logging.getLogger(__name__)
logger.setLevel(5)
logger.addHandler(gz_rotfile_loghandler)
logger.addHandler(simplefile_loghandler)
logger.addHandler(stream_loghandler)
However, pylint complains (and i agree) that methods defined in a class should either be static methods, or follow the naming conventions for first parameters (e.g. gz_log_rotator(self, dest)), which is not how the function is used, and would be much more confusing.
Fun Fact
During this process i've also discovered that instances of classmethod and staticmethod are not in and of themselves callable (???). While a method defined in a class namespace is callable both within and without, classmethods and staticmethods are only callable when accessed through their class (at which point they refer to the underlying function, not the classmethod/staticmethod object)
>>> class Thing:
... global one_, two_, three_
... def one(self):
... print('one')
... #classmethod
... def two(cls):
... print('two')
... #staticmethod
... def three():
... print('three')
... one_, two_, three_ = one, two, three
...
>>> Thing.one()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: one() missing 1 required positional argument: 'self'
>>> Thing.two()
two
>>> Thing.three()
three
>>> # all as expected
>>> one_()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: one() missing 1 required positional argument: 'self'
>>> # so far so good
>>> two_()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'classmethod' object is not callable
>>> # what?
>>> three_()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'staticmethod' object is not callable
>>> # ???
My Question
Is there a better way to hold these variables without polluting my namespace?
The code I have works correctly, but it makes me feel a little unclean. I could define a function that would only be called once and then immediately call it, but then I either lose references to everything I don't return, or i'm back to polluting the global namespace. I could just make everything _hidden, but I feel like they should be logically grouped. I could make _Logging a bona fide class, put all of my stuff in an __init__ function and tack all my little variables onto self, but that also feels inelegant. I could create another file for this, but so far I've gotten by with everything held in the same file. The only other option that seemed palatable is to make the two functions staticmethods and only refer to them through our class (i.e. _Logging.gz_log_namer), but it would seem that is also impossible.
>>> class Thing:
... #staticmethod
... def say_hello():
... print('hello!')
... Thing.say_hello()
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in Thing
AttributeError: type object 'Thing' has no attribute 'say_hello'
>>>
As it stands, the best option I see is to use the selfless methods.
you can create a new class that inherit from staticmethod class, and add __call__ method to the class.
for example:
class callablestatic(staticmethod):
def __init__(self, func):
super().__init__(func)
self.func = func
def __call__(self, *args, **kwargs):
# the __call__ method allows you to call the class instance
return self.func(*args, **kwargs)
then use it in your class:
class Thing:
#callablestatic
def hello(name):
print(f"hello {name}")
hello("John") # works
but better create new file and import it as a module
Sorry for answering 2 years later, but this could help someone.
You could make your methods static, and create another static method (ex. init), calling it right after initializing the class. Then use setattr to keep the references to your variables.
For setting multiple class variables, you can use
[setattr(Class, name, value) for name,value in locals().items()]
inside the method.
Full code:
class _Logging:
'''A little namespace for our logging facilities. Don't try to instantiate
it: all it does is group together some logging objects and keep them out of
the global namespace'''
def __init__(self):
raise TypeError("that's not how this works...")
#staticmethod
def gz_log_rotator(source, dest):
'''accept a source filename and a destination filename. copy source to
dest and add gzip compression. for use with
logging.handlers.RotatingFileHandler.rotator.'''
with gzip.open(dest, 'wb', 1) as ofile, open(source, 'rb') as ifile:
ofile.write(ifile.read())
os.remove(source)
#staticmethod
def gz_log_namer(name):
'''accept a filename, and return it with ".gz" appended. for use with
logging.handlers.RotatingFileHandler.namer.'''
return name + ".gz"
#staticmethod
def init():
global logger
fmtr = logging.Formatter(
'[%(asctime)s:%(name)s:%(thread)05d:%(levelname)-8s] %(message)s')
gz_rotfile_loghandler = logging.handlers.RotatingFileHandler(
'%s.log' % __name__, mode='a', maxBytes=(1024**2 * 20), backupCount=3)
gz_rotfile_loghandler.setLevel(5)
gz_rotfile_loghandler.setFormatter(fmtr)
gz_rotfile_loghandler.rotator = _Logging.gz_log_rotator
gz_rotfile_loghandler.namer = _Logging.gz_log_namer
simplefile_loghandler = logging.FileHandler(
'%s.simple.log' % __name__, mode='w')
simplefile_loghandler.setLevel(15)
simplefile_loghandler.setFormatter(fmtr)
stream_loghandler = logging.StreamHandler()
stream_loghandler.setLevel(25)
stream_loghandler.setFormatter(fmtr)
logger = logging.getLogger(__name__)
logger.setLevel(5)
logger.addHandler(gz_rotfile_loghandler)
logger.addHandler(simplefile_loghandler)
logger.addHandler(stream_loghandler)
[setattr(_Logging, name, value) for name,value in locals().items()]
_Logging.init()
I wonder why the python magic method (str) always looking for the return statement rather a print method ?
class test:
def __init__(self):
print("constructor called")
def __call__(self):
print("callable")
def __str__(self):
return "string method"
obj=test() ## print constructor called
obj() ### print callable
print(obj) ## print string method
my question is why i can't use something like this inside the str method
def __str__(self):
print("string method")
This is more to enable the conversion of an object into a str - your users don't necessary want all that stuff be printed into the terminal whenever they want to do something like
text = str(obj_instance)
They want text to contain the result, not printed out onto the terminal.
Doing it your way, the code would effectively be this
text = print(obj_instance)
Which is kind of nonsensical because the result of print isn't typically useful and text won't contain the stream of text that was passed into str type.
As you already commented (but since deleted), not providing the correct type for the return value will cause an exception to be raised, for example:
>>> class C(object):
... def __str__(self):
... return None
...
>>> str(C())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __str__ returned non-string (type NoneType)
>>>
Because __str__() is used when you print the object, so the user is already calling print which needs the String that represent the Object - as a variable to pass back to the user's print
In the example you provided above, if __str__ would print you would get:
print(obj)
translated into:
print(print("string method"))
Does Python have extension methods like C#? Is it possible to call a method like:
MyRandomMethod()
on existing types like int?
myInt.MyRandomMethod()
You can add whatever methods you like on class objects defined in Python code (AKA monkey patching):
>>> class A(object):
>>> pass
>>> def stuff(self):
>>> print self
>>> A.test = stuff
>>> A().test()
This does not work on builtin types, because their __dict__ is not writable (it's a dictproxy).
So no, there is no "real" extension method mechanism in Python.
It can be done with Forbidden Fruit (https://pypi.python.org/pypi/forbiddenfruit)
Install forbiddenfruit:
pip install forbiddenfruit
Then you can extend built-in types:
>>> from forbiddenfruit import curse
>>> def percent(self, delta):
... return self * (1 + delta / 100)
>>> curse(float, 'percent', percent)
>>> 1.0.percent(5)
1.05
Forbidden Fruit is fundamentally dependent on the C API, it works only on cpython implementations and won’t work on other python implementations, such as Jython, pypy, etc.
not sure if that what you're asking but you can extend existing types and then call whatever you like on the new thing:
class int(int):
def random_method(self):
return 4 # guaranteed to be random
v = int(5) # you'll have to instantiate all you variables like this
v.random_method()
class int(int):
def xkcd(self):
import antigravity
print(42)
>>>v.xkcd()
Traceback (most recent call last):
File "<pyshell#81>", line 1, in <module>
v.xkcd()
AttributeError: 'int' object has no attribute 'xkcd'
c = int(1)
>>> c.random_method()
4
>>> c.xkcd()
42
hope that clarifies your question
The following context manager adds the method like Forbidden Fruit would without the limitations of it. Besides that it has the additional benefit of removing the extension method afterwards:
class extension_method:
def __init__(self, obj, method):
method_name = method.__name__
setattr(obj, method_name, method)
self.obj = obj
self.method_name = method_name
def __enter__(self):
return self.obj
def __exit__(self, type, value, traceback):
# remove this if you want to keep the extension method after context exit
delattr(self.obj, self.method_name)
Usage is as follows:
class C:
pass
def get_class_name(self):
return self.__class__.__name__
with extension_method(C, get_class_name):
assert hasattr(C, 'get_class_name') # the method is added to C
c = C()
print(c.get_class_name()) # prints 'C'
assert not hasattr(C, 'get_class_name') # the method is gone from C
I've had great luck with the method described here:
http://mail.python.org/pipermail/python-dev/2008-January/076194.html
I have no idea if it works on builtins though.
Another option is to override the meta-class. This allows you to, among other things, specify functions that should exist in all classes.
This article starts to discuss it:
http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html