I have a multi-module package in python. One of the modules is essentially a command line application. We'll call that one the "top level" module. The other module has three classes in it, which are essentially the backend of the application.
The toplevel module, in the init for it's class, does logging.basicConfig to log debug to file, then adds a console logger for info and above. The backend classes just use getLogger(classname), because when the application run in full, the backend will be called by the top level command line frontend, so logging will already be configured.
In the Test class (subclassed from unittest.TestCase and run via nose), I simply run testfixtures.LogCapture() in setup, and testfixtures.LogCapture.uninstall_all() in tearDown, and all the logging is captured just fine, no effort.
In the backend test file, I tried to do the same thing. I run testfixtures.LogCapture in the setup, uninstall_all in the teardown. However, all the "INFO" level logmessages still print when I'm running unittests for the backend.
Any help on
1) why log capture works for the frontend but not backend
2) an elegant way to be able to log and capture logs in my backend class without explictly setting up logging in those files.
would be amazing.
I fixed the same issue by setting 'disable_existing_loggers' to False when reconfiguring the Logger: the previous logger was disabled and it was preventing it from propagating the logs to the RootLogger.
Related
I am working on a test framework. Each test is launched as a new python multiprocessing process.
There is one master log file and individual log files corresponding to each test.
There is a master logger created at the launch of framework code and a new logger created in each test process. Test loggers log to both - it's own log file and master log file.
There are multiple libraries that can be used by any of the test.
Currently there is no logging done in library function. In order to add logging to the library functions, logger object needs to passed as a parameter to this library function. To achieve this, every function signature in library modules and function call will have to be modified, which is not practical.
As I understand, I cannot have module level logger because module level logger will log to different file for each module and not for each test process.
Can you suggest a solution where I don't have to pass log objects around function and log statements would log to the right file based on which process is calling the function?
The threading module has a get_ident member which could be used to index some logger dictionary, something like;
from threading import get_ident
loggers[get_ident()].logError('blah blah blah')
However, once you have all of this test logging throughout your libraries, how will that impact your production performance?
Updated:
Turns out, this is not a function of cron. I get the same behavior when running the script from the command line, if it in fact has a record to process and communicates with ElasticSearch.
I have a cron job that runs a python script which uses pyelasticsearch to index some documents in an ElasticSearch instance. The script works fine from the command line, but when run via cron, it results in this error:
No handlers could be found for logger "elasticsearch.trace"
Clearly there's some logging configuration issue that only crops up when run under cron, but I'm not clear what it is. Any insight?
I solved this by explicitly configuring a handler for the elasticsearch.trace logger, as I saw in examples from the pyelasticsearch repo.
After importing pyelasticsearch, set up a handler like so:
tracer = logging.getLogger('elasticsearch.trace')
tracer.setLevel(logging.INFO)
tracer.addHandler(logging.FileHandler('/tmp/es_trace.log'))
I'm not interested in keeping the trace logs, so I used the near-at-hand Django NullHandler.
from django.utils.log import NullHandler
tracer = logging.getLogger('elasticsearch.trace')
tracer.setLevel(logging.INFO)
tracer.addHandler(NullHandler())
I am trying to setup logging when using IPython parallel. Specifically, I would like to redirect log messages from the engines to the client. So, rather than each of the engines logging individually to their own log files, as in IPython.parallel - can I write my own log into the engine logs?, I am looking for something like How should I log while using multiprocessing in Python?
Based on reviewing the IPython code base, I have the impression that the way to do this would be to register a zmq.log.hander.PUBHandler with the logging module (see documentation in iploggerapp.py). I have tried this in various ways, but none seem to work. I also tried to register a logger via IPython.parallel.util. connect_engine_logger, but this also does not appear to do anything.
update
I have made some progress on this problem. If I specify in ipengine_config c.IPEngineApp.log_url, then the logger of the IPython application has the appropriate EnginePubHandler. I checked this via
%%px
from IPython.config import Application
log = Application.instance().log
print(log.handlers)
Which indicated that the application logger has an EnginePUBHandler for each engine. Next, I can start the iplogger app in a separate terminal and see the log messages from each engine.
However, What I would like to achieve is to see these log messages in the notebook, rather than in a separate terminal. I have tried starting iplogger from within the notebook via a system call, but this crashes.
I have the following in my tasks.py
from celery.utils.log import get_task_logger
logger = get_task_logger("celery.task")
I have setup logging for celery.task in my settings.py and all the log from my tasks.py file are properly logged to file.
I have Django modules. These modules can be called directly from Django or by a Celery task. What I want is, if the module is called by Django, then logs should go to Django log file. If the module is called by a task, the log should go to celery task logger.
Example:
# tasks.py
from app.foo import MyFoo
#task(name="bar")
def boo():
MyFoo.someFoo()
# app.foo.py
log = logging.getLogger(__name__)
I want the log messages inside MyFoo to go to celery log when run by a worker task.
Any ideas?
You should be able to configure loggers separately for Django process and Celery process, as if by definition they run in separate processes.
Actually I am surprised that the log output does not go to a separate log files; thus maybe it would make sense to you expose what kind of logging configurations you have already in-place.
Where Python logging output goes is defined by logging handlers.
Configuring logger for Django: https://docs.djangoproject.com/en/dev/topics/logging/
For Celery, there doesn't seem to be straightforward way to override logging settings. They seem to give a celery.signals.setup_logging where you could set a breakpoint, use logging API to reconfigure logging handlers to go to a separate log file.
http://docs.celeryproject.org/en/latest/configuration.html#logging
Alternatively, you can just pull out different logger object on a task level. When you execute a task you know whether it is executed from Django (eager) or Celery.
I have never done this myself, but apparently Celery tasks expose the task running context as self.request parameter (nothing to do with Python classes).
http://docs.celeryproject.org/en/latest/userguide/tasks.html#context
So at the beginning of your task function you could switch between the loggers:
# Define web_worker_logger
# Define task_logger
#task(name='example')
def moobar():
logger = web_worker_logger if self.request.is_eager else task_logger
I'm looking at how to log to syslog from within my Python app, and I found there are two ways of doing it:
Using syslog.syslog() routines
Using the logger module SysLogHandler
Which is the best option to use, advantages/disadvantages of each one, etc, because I really don't know which one should I use.
syslog.syslog() can only be used to send messages to the local syslogd. SysLogHandler can be used as part of a comprehensive, configurable logging subsystem, and can log to remote machines.
The logging module is a more comprehensive solution that can potentially handle all of your log messages, and is very flexible. For instance, you can setup multiple handers for your logger and each can be set to log at a different level. You can have a SysLogHandler for sending errors to syslog, and a FileHandler for debugging logs, and an SMTPHandler to email the really critical messages to ops. You can also define a hierarchy of loggers within your modules, and each one has its own level so you can enable/disable messages from specific modules, such as:
import logging
logger = logging.getLogger('package.stable_module')
logger.setLevel(logging.WARNING)
And in another module:
import logging
logger = logging.getLogger('package.buggy_module')
logger.setLevel(logging.DEBUG)
The log messages in both of the these modules will be sent, depending on the level, to the 'package' logger and ultimately to the handlers you've defined. You can also add handlers directly to the module loggers, and so on. If you've followed along this far and are still interested, then I recommend jumping to the logging tutorial for more details.
So far, there is a disadvantage in logging.handlers.SysLogHander which is not mentioned yet. That is I can't set options like LOG_ODELAY or LOG_NOWAIT or LOG_PID. On the other hands, LOG_CONS and LOG_PERROR can be achieved with adding more handlers, and LOG_NDELAY is already set by default, because the connection opens when the handler is instantiated.