A bit of background
I'm writing a python module for my own use, and I'm using Python's logging module. There are handlers and formatters and even a pair of functions I create that (for the most part) won't be used anywhere else. However, I still want to be able to access and modify these variables elsewhere (for instance, other closely-coupled modules or scripts)
A simple namespace
What I'm currently doing is using a class definition to group all of my variables together, like this:
class _Logging:
'''A little namespace for our logging facilities. Don't try to instantiate
it: all it does is group together some logging objects and keep them out of
the global namespace'''
global logger
def __init__(self):
raise TypeError("that's not how this works...")
def gz_log_rotator(source, dest):
'''accept a source filename and a destination filename. copy source to
dest and add gzip compression. for use with
logging.handlers.RotatingFileHandler.rotator.'''
with gzip.open(dest, 'wb', 1) as ofile, open(source, 'rb') as ifile:
ofile.write(ifile.read())
os.remove(source)
def gz_log_namer(name):
'''accept a filename, and return it with ".gz" appended. for use with
logging.handlers.RotatingFileHandler.namer.'''
return name + ".gz"
fmtr = logging.Formatter(
'[%(asctime)s:%(name)s:%(thread)05d:%(levelname)-8s] %(message)s')
gz_rotfile_loghandler = logging.handlers.RotatingFileHandler(
'%s.log' % __name__, mode='a', maxBytes=(1024**2 * 20), backupCount=3)
gz_rotfile_loghandler.setLevel(5)
gz_rotfile_loghandler.setFormatter(fmtr)
gz_rotfile_loghandler.rotator = gz_log_rotator
gz_rotfile_loghandler.namer = gz_log_namer
simplefile_loghandler = logging.FileHandler(
'%s.simple.log' % __name__, mode='w')
simplefile_loghandler.setLevel(15)
simplefile_loghandler.setFormatter(fmtr)
stream_loghandler = logging.StreamHandler()
stream_loghandler.setLevel(25)
stream_loghandler.setFormatter(fmtr)
logger = logging.getLogger(__name__)
logger.setLevel(5)
logger.addHandler(gz_rotfile_loghandler)
logger.addHandler(simplefile_loghandler)
logger.addHandler(stream_loghandler)
However, pylint complains (and i agree) that methods defined in a class should either be static methods, or follow the naming conventions for first parameters (e.g. gz_log_rotator(self, dest)), which is not how the function is used, and would be much more confusing.
Fun Fact
During this process i've also discovered that instances of classmethod and staticmethod are not in and of themselves callable (???). While a method defined in a class namespace is callable both within and without, classmethods and staticmethods are only callable when accessed through their class (at which point they refer to the underlying function, not the classmethod/staticmethod object)
>>> class Thing:
... global one_, two_, three_
... def one(self):
... print('one')
... #classmethod
... def two(cls):
... print('two')
... #staticmethod
... def three():
... print('three')
... one_, two_, three_ = one, two, three
...
>>> Thing.one()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: one() missing 1 required positional argument: 'self'
>>> Thing.two()
two
>>> Thing.three()
three
>>> # all as expected
>>> one_()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: one() missing 1 required positional argument: 'self'
>>> # so far so good
>>> two_()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'classmethod' object is not callable
>>> # what?
>>> three_()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'staticmethod' object is not callable
>>> # ???
My Question
Is there a better way to hold these variables without polluting my namespace?
The code I have works correctly, but it makes me feel a little unclean. I could define a function that would only be called once and then immediately call it, but then I either lose references to everything I don't return, or i'm back to polluting the global namespace. I could just make everything _hidden, but I feel like they should be logically grouped. I could make _Logging a bona fide class, put all of my stuff in an __init__ function and tack all my little variables onto self, but that also feels inelegant. I could create another file for this, but so far I've gotten by with everything held in the same file. The only other option that seemed palatable is to make the two functions staticmethods and only refer to them through our class (i.e. _Logging.gz_log_namer), but it would seem that is also impossible.
>>> class Thing:
... #staticmethod
... def say_hello():
... print('hello!')
... Thing.say_hello()
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in Thing
AttributeError: type object 'Thing' has no attribute 'say_hello'
>>>
As it stands, the best option I see is to use the selfless methods.
you can create a new class that inherit from staticmethod class, and add __call__ method to the class.
for example:
class callablestatic(staticmethod):
def __init__(self, func):
super().__init__(func)
self.func = func
def __call__(self, *args, **kwargs):
# the __call__ method allows you to call the class instance
return self.func(*args, **kwargs)
then use it in your class:
class Thing:
#callablestatic
def hello(name):
print(f"hello {name}")
hello("John") # works
but better create new file and import it as a module
Sorry for answering 2 years later, but this could help someone.
You could make your methods static, and create another static method (ex. init), calling it right after initializing the class. Then use setattr to keep the references to your variables.
For setting multiple class variables, you can use
[setattr(Class, name, value) for name,value in locals().items()]
inside the method.
Full code:
class _Logging:
'''A little namespace for our logging facilities. Don't try to instantiate
it: all it does is group together some logging objects and keep them out of
the global namespace'''
def __init__(self):
raise TypeError("that's not how this works...")
#staticmethod
def gz_log_rotator(source, dest):
'''accept a source filename and a destination filename. copy source to
dest and add gzip compression. for use with
logging.handlers.RotatingFileHandler.rotator.'''
with gzip.open(dest, 'wb', 1) as ofile, open(source, 'rb') as ifile:
ofile.write(ifile.read())
os.remove(source)
#staticmethod
def gz_log_namer(name):
'''accept a filename, and return it with ".gz" appended. for use with
logging.handlers.RotatingFileHandler.namer.'''
return name + ".gz"
#staticmethod
def init():
global logger
fmtr = logging.Formatter(
'[%(asctime)s:%(name)s:%(thread)05d:%(levelname)-8s] %(message)s')
gz_rotfile_loghandler = logging.handlers.RotatingFileHandler(
'%s.log' % __name__, mode='a', maxBytes=(1024**2 * 20), backupCount=3)
gz_rotfile_loghandler.setLevel(5)
gz_rotfile_loghandler.setFormatter(fmtr)
gz_rotfile_loghandler.rotator = _Logging.gz_log_rotator
gz_rotfile_loghandler.namer = _Logging.gz_log_namer
simplefile_loghandler = logging.FileHandler(
'%s.simple.log' % __name__, mode='w')
simplefile_loghandler.setLevel(15)
simplefile_loghandler.setFormatter(fmtr)
stream_loghandler = logging.StreamHandler()
stream_loghandler.setLevel(25)
stream_loghandler.setFormatter(fmtr)
logger = logging.getLogger(__name__)
logger.setLevel(5)
logger.addHandler(gz_rotfile_loghandler)
logger.addHandler(simplefile_loghandler)
logger.addHandler(stream_loghandler)
[setattr(_Logging, name, value) for name,value in locals().items()]
_Logging.init()
Related
I am trying to call a method on an other class and give the called class a reference of the current class along with some other parameters. But somehow it takes the self given as a parameter as the self of the called class.
Let me show you:
import os, sys
from wsPart import wsPart
class thermo(wsPart):
functional = False ## see line 8
file = '/sys/bus/w1/devices/28-00000833e8ff/w1_slave'
def __init__(self, name, logger):
super().__init__(name, logger)
functional = True
def read(self):
fileobject = open(self.file)
filecontent = fileobject.read()
fileobject.close()
self.logger.writeLog(self,"Completed Meassurement") ##Problem on this line
return filecontent
So I call the class logger and the method writeLog on it. Giving the Parameters message and a reference of the class thermo (self).
import datetime
from wsPart import wsPart
class logger():
logfile = "/var/log/wheaterstation.log"
name = "Logger"
def writeLog(self, sender, message):
conn = open(self.logfile, "w")
now = str(datetime.datetime.now().isoformat())
conn.write("[" + now + "]" + " (" + sender.getName() + "): " + message + "\n") ##Problem on this line
conn.close()
As you can see I put the parameters self because its a method that belongs to a class, the sender should be the reference to the class thermo that was passed as self in the thermo class. Lastly there is the message which was passed in the thermo class as well.
But this just gives me the error:
Traceback (most recent call last):
File "scrLib/wsControl.py", line 61, in <module>
controller = controller()
File "scrLib/wsControl.py", line 22, in __init__
self.thermo = thermo("Thermometer", logger)
File "/home/joco/git/wheaterstation/scrLib/thermo.py", line 10, in __init__
super().__init__(name, logger)
File "/home/joco/git/wheaterstation/scrLib/wsPart.py", line 8, in __init__
self.logger.writeLog(self, "created")
TypeError: writeLog() missing 1 required positional argument: 'message'
So it seems that the self parameter which was passed in the thermo class is interpeted as the self of the class logger which gets it all mixed up.
Can you guys help me here?
The full code + additional comments can be viewed Here
Edit:
Both the logger and the thermo class get initialized in the file wsPart.py:
class controller():
name = ""
logger = None
thermo = None
dbConnector = None
def __init__(self):
##THis created the controller and all the other objects
self.name = "Controller"
##Create Objects
self.logger = logger()
self.logger.writeLog(self,"logger created") ##This line Works
self.thermo = thermo("Thermometer", logger)
self.dbConnector = dbConnector("DBConnector",logger)
yes, bad idea to call the instance and the class name the same. Here:
self.logger = logger()
self.logger.writeLog(self,"logger created") ##This line Works
self.thermo = thermo("Thermometer", logger)
self.dbConnector = dbConnector("DBConnector",logger)
You're passing the class itself to your constructors. So the methods are seen as static/expect one more parameter. You need to change the 2 last lines to pass the instance you just created:
self.thermo = thermo("Thermometer", self.logger)
self.dbConnector = dbConnector("DBConnector", self.logger)
more importantly, you need to use different names for classes and instances of the same objects to avoid that confusion (python convention for class names is starting each word with upper case (camelcase) ex: Logger. Other languages don't use that convention, but python is a lot about conventions).
With a different name you'd have gotten a NameError exception and you would have fixed the error yourself.
Aside: don't "initialize" members like this in the class definition:
name = ""
logger = None
thermo = None
dbConnector = None
those are creating class members, not instance members. Remove those, and let __init__ create instance members like you're currently doing. __init__ is called no matter what, and those lines above just add to the confusion (except for some corner cases, only constants should be declared that way)
Totally unrelated but code in comments is unreadable so I post this as an answer:
this does not work as you seem to expect:
class Whatever():
functional = False ## see line 8
def __init__(self, name, logger):
super().__init__(name, logger)
functional = True
Python has no "implied this" sor here in __init__ you're not creating an instance attribute but a local variable. You want self.functional = True
Make sure you close files
def read(self):
fileobject = open(self.file)
filecontent = fileobject.read()
fileobject.close()
If anything wrong happens between open() and fileobject.close(), the file is not garanteed to be properly closed. You want eiher a try/finally block ie
f = open(path)
try:
do_something_with(f)
finally:
f.close()
or much better a with block:
with open(path) as f:
do_something_with(f)
which will ensure the file is closed whatever happens.
write mode truncates the file
def writeLog(self, sender, message):
conn = open(self.logfile, "w")
now = str(datetime.datetime.now().isoformat())
conn.write("[" + now + "]" + " (" + sender.getName() + "): " + message + "\n") ##Problem on this line
conn.close()
as documented, opening a file in write mode truncates the file. You probably want the "append" mode instead here.
Don't reinvent the squared wheel when there's a round one already
Logging is not as trivial as just writing to a file (concurrency issues, need to send the log message to some other destination, logging levels etc), and even if you don't need more (at least the moment) your solution is quite inefficient (opening a file is expensive).
Python has a very comprehensive logging package in it's standard lib. I wholefully agree that it requires a bit of learning to configure and use properly but that's still a huge win compared to the time you'll spend trying to make a naive half-backed custom implementation works properly on production, AND this is a knowledge that you will need for just any serious project anyway.
In the documentation on instance methods it states that:
Methods also support accessing (but not setting) the arbitrary function attributes on the underlying function object.
But I can't seem to be able to verify that restriction. I tried setting both an arbitrary value and one of the "Special Attributes" of functions:
class cls:
def foo(self):
f = self.foo.__func__
f.a = "some value" # arbitrary value
f.__doc__ = "Documentation"
print(f.a, f.__doc__)
When executed, no errors are produced and the output is as expected:
cls().foo() # prints out f.a, f.__doc__
What is it that I'm misunderstanding with the documentation?
You are misunderstanding what is being said. It says that you can access but not set the attributes of the underlying function object from the method!
>>> class Foo:
... def foo(self):
... self.foo.__func__.a = 1
... print(self.foo.a)
... self.foo.a = 2
...
>>> Foo().foo()
1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in foo
AttributeError: 'method' object has no attribute 'a'
Note how foo.a is updated when you set it on the __func__ value, but you cannot set it directly using self.foo.a = value.
So the function object can be modified as you please, the method wrapper only provides read-only access to the attributes on the underlying function.
Python 2.7
I have my own exception:
class NoSourceFileError(Exception):
def __init__(self, logger, massage):
Exception.__init__(self, massage)
logger.logger.info(massage)
And it's calls with:
...
else:
raise NoSourceFileError('ERROR: can not find file %s for %s' % (add_file, add_name))
Problem here as you see - that I pass two variables (add_file, add_name) - but __init__ can accept only one var (message).
How can I pass both of them?
I tried play with *args - but can't make it work.
Logger - my additional class for logging.
The issue is not the string, which is fine. It is that you are passing logger into the exception.
Remove the logger parameter (and usage) and it will work:
>>> class NoSourceFileError(Exception):
... def __init__(self, message):
... Exception.__init__(self, message)
...
>>> raise NoSourceFileError('ERROR: can not find file %s for %s' % ('x', 'y'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
__main__.NoSourceFileError: ERROR: can not find file x for y
>>>
I'm not sure what logger is meant to be. But if you want to use it, you need to pass it as an argument to the exception when raising it.
On string formatting, it on it's own will always count as a single argument assuming it is done right. So can't count for more than 1.
As #werkritter has said, if you do want to use logger without having to pass it in as a parameter - define it globally. I will assume that it is meant to log errors, in which case it would make sense to have it defined globally anyway.
I try to create interface with #staticmethod and #classmethod. Declaring class method is simple. But I can't find the correct way to declare static method.
Consider class interface and its implementation:
#!/usr/bin/python3
from zope.interface import Interface, implementer, verify
class ISerializable(Interface):
def from_dump(slice_id, intex_list, input_stream):
'''Loads from dump.'''
def dump(out_stream):
'''Writes dump.'''
def load_index_list(input_stream):
'''staticmethod'''
#implementer(ISerializable)
class MyObject(object):
def dump(self, out_stream):
pass
#classmethod
def from_dump(cls, slice_id, intex_list, input_stream):
return cls()
#staticmethod
def load_index_list(stream):
pass
verify.verifyClass(ISerializable, MyObject)
verify.verifyObject(ISerializable, MyObject())
verify.verifyObject(ISerializable, MyObject.from_dump(0, [], 'stream'))
Output:
Traceback (most recent call last):
File "./test-interface.py", line 31, in <module>
verify.verifyClass(ISerializable, MyObject)
File "/usr/local/lib/python3.4/dist-packages/zope/interface/verify.py", line 102, in verifyClass
return _verify(iface, candidate, tentative, vtype='c')
File "/usr/local/lib/python3.4/dist-packages/zope/interface/verify.py", line 97, in _verify
raise BrokenMethodImplementation(name, mess)
zope.interface.exceptions.BrokenMethodImplementation: The implementation of load_index_list violates its contract
because implementation doesn't allow enough arguments.
How should I correctly declare static method in this interface?
Obviously the verifyClass does not understand either classmethod or staticmethod properly. The problem is that in Python 3, if you do getattr(MyObject, 'load_index_list') in Python 3, you get a bare function, and verifyClass thinks it is yet another unbound method, and then expects that the implicit self be the first argument.
The easiest fix is to use a classmethod there instead of a staticmethod.
I guess someone could also do a bug report.
I know how to override an object's getattr() to handle calls to undefined object functions. However, I would like to achieve the same behavior for the builtin getattr() function. For instance, consider code like this:
call_some_undefined_function()
Normally, that simply produces an error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'call_some_undefined_function' is not defined
I want to override getattr() so that I can intercept the call to "call_some_undefined_function()" and figure out what to do.
Is this possible?
I can only think of a way to do this by calling eval.
class Global(dict):
def undefined(self, *args, **kargs):
return u'ran undefined'
def __getitem__(self, key):
if dict.has_key(self, key):
return dict.__getitem__(self, key)
return self.undefined
src = """
def foo():
return u'ran foo'
print foo()
print callme(1,2)
"""
code = compile(src, '<no file>', 'exec')
globals = Global()
eval(code, globals)
The above outputs
ran foo
ran undefined
You haven't said why you're trying to do this. I had a use case where I wanted to be capable of handling typos that I made during interactive Python sessions, so I put this into my python startup file:
import sys
import re
def nameErrorHandler(type, value, traceback):
if not isinstance(value, NameError):
# Let the normal error handler handle this:
nameErrorHandler.originalExceptHookFunction(type, value, traceback)
name = re.search(r"'(\S+)'", value.message).group(1)
# At this point we know that there was an attempt to use name
# which ended up not being defined anywhere.
# Handle this however you want...
nameErrorHandler.originalExceptHookFunction = sys.excepthook
sys.excepthook = nameErrorHandler
Hopefully this is helpful for anyone in the future who wants to have a special error handler for undefined names... whether this is helpful for the OP or not is unknown since they never actually told us what their intended use-case was.