I'm trying to use standard Python logging module alongside with gevent. I have monkey patched threading and I expect logging to work with my app:
import gevent.monkey
gevent.monkey.patch_all()
import logging
logger = logging.getLogger()
fh = logging.FileHandler('E:\\spam.log')
fh.setLevel(logging.DEBUG)
logger.addHandler(fh)
def foo():
logger.debug('Running in foo')
gevent.sleep(0)
logger.debug('Explicit context switch to foo again')
def bar():
logger.debug('Explicit context to bar')
gevent.sleep(0)
logger.debug('Implicit context switch back to bar')
gevent.joinall([
gevent.spawn(foo),
gevent.spawn(bar),
])
Unfortunately E:\spam.log is empty and I can see no output to the console. It seems that either I haven't configured logging properly or gevent does not support it at all (which I don't believe, because gevent documentation says that it does). So, how can I log at gevent app?
You haven't configured it correctly. You need to set the DEBUG level on the logger rather than the handler, otherwise the logger's default level (WARNING) causes your debug messages to be dropped. Try doing a
logger.setLevel(logging.DEBUG)
and it should work.
Related
I have a question regarding logging module in Python.
If I instantiate a logger and set its level to INFO without adding any handlers, then I expect that info messages will be printed on the screen.
However, in reality, warning messages are printed but info messages are not.
After adding explicitly a handler, both levels are printed.
Precisely, the following small script:
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.info("This is an info message")
logger.warning("This is a warning message")
print("*** Add a stream handler explicitly")
handler = logging.StreamHandler()
logger.addHandler(handler)
logger.info("This is an info message")
logger.warning("This is a warning message")
gives output
This is a warning message
*** Add a stream handler explicitly
This is an info message
This is a warning message
(checked in Python 3.7.6 and 3.8.2).
I would expect that either no messages are printed without a handler, or both levels are printed after setting level to INFO.
I suggest you use loguru. It's a really simple to use tool and for me it's very intuitive.
You can add a default logger just using this line:
from loguru import logger
If you want to change the logging level, just setup the logger like this:
logger.remove() # This disables the default logger
logger.add(sys.stderr, level="INFO") # This adds a default logger (format and colors included) but at the logging level you need
And that's it. All the scripts that use the loguru logger in your code will be set up like this. By default it outputs to stderr but you can change it however you need.
I am using wsgi server to spawn the servers for my web application. I am having problem with information logging.
This is how I am running the app
from gevent import monkey; monkey.patch_all()
from logging.handlers import RotatingFileHandler
import logging
from app import app # this imports app
# create a file to store weblogs
log = open(ERROR_LOG_FILE, 'w'); log.seek(0); log.truncate();
log.write("Web Application Log\n"); log.close();
log_handler = RotatingFileHandler(ERROR_LOG_FILE, maxBytes =1000000, backupCount=1)
formatter = logging.Formatter(
"[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s"
)
log_handler.setFormatter(formatter)
app.logger.setLevel(logging.DEBUG)
app.logger.addHandler(log_handler)
# run the application
server= wsgi.WSGIServer(('0.0.0.0', 8080), app)
server.serve_forever()
However, on running the application it is not logging anything. I guess it must be because of WSGI server because app.logger works in the absence of WSGI. How can I log information when using WSGI?
According to the gevent uwsgi documentation you need to pass your log handler object to the WSGIServer object at creation:
log – If given, an object with a write method to which request (access) logs will be written. If not given, defaults to sys.stderr. You may pass None to disable request logging. You may use a wrapper, around e.g., logging, to support objects that don’t implement a write method. (If you pass a Logger instance, or in general something that provides a log method but not a write method, such a wrapper will automatically be created and it will be logged to at the INFO level.)
error_log – If given, a file-like object with write, writelines and flush methods to which error logs will be written. If not given, defaults to sys.stderr. You may pass None to disable error logging (not recommended). You may use a wrapper, around e.g., logging, to support objects that don’t implement the proper methods. This parameter will become the value for wsgi.errors in the WSGI environment (if not already set). (As with log, wrappers for Logger instances and the like will be created automatically and logged to at the ERROR level.)
so you should be able to do wsgi.WSGIServer(('0.0.0.0', 8080), app, log=app.logger)
You can log like this -
import logging
import logging.handlers as handlers
.
.
.
logger = logging.getLogger('MainProgram')
logger.setLevel(10)
logHandler = handlers.RotatingFileHandler(filename.log,maxBytes =1000000, backupCount=1)
logger.addHandler(logHandler)
logger.info("Logging configuration done")
.
.
.
# run the application
server= wsgi.WSGIServer(('0.0.0.0', 8080), app, log=logger)
server.serve_forever()
How does one turn on celery logging programmatically?
From the terminal, this works fine:
celery worker -l DEBUG
When I call get_task_logger(__name__).debug('hello'), I can see the message come up in the terminal. (stdout and stderr are being displayed) I can even import logging and call logger.info('hi') and see that too. (both work)
However, while developing a task, I prefer to use a test module and call the task function directly rather than firing up a whole worker. But I can't see the log messages. I understand that Celery is redirecting everything to its internal apparatus, but I want to see the log messages on the stdout too.
How do I tell Celery to send a copy of the log messages back to stdout?
I've read a bunch of online articles about logging but it seems that a number of logging-related configuration vars from celery have been deprecated and it's unclear to me from the docs what is the supported path today.
Here is an example module that creates a celery object and attempts to log output. Nothing shows in the terminal.
example mymodule.py
from celery import Celery
import logging
from celery.utils.log import get_task_logger
app = Celery('test')
app.config_from_object('myfile', True)
get_task_logger(__name__).warn('hello world')
logging.getLogger(__name__).warn('hello world 2')
EDIT
I know that I can add a handler to redirect some of the output back to the terminal by adding a handler
log = get_task_logger(__name__)
h = logging.StreamHandler(sys.stdout)
log.addHandler(h)
But is there a "Celery way" to do this? Maybe one that lets me also have the Celery formatted lines of text.
[2014-03-02 15:51:32,949: WARNING] hello world
I have been looking at the same issue...
What seems to work best is to use the signal handler, according to http://docs.celeryproject.org/en/latest/userguide/signals.html#after-setup-logger
In your celery.py file use:
from celery.signals import after_setup_logger
import logging
#after_setup_logger.connect()
def logger_setup_handler(logger, **kwargs ):
my_handler = MyLogHandler()
my_handler.setLevel(logging.DEBUG)
my_formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') #custom formatter
my_handler.setFormatter(my_formatter)
logger.addHandler(my_handler)
logging.info("My log handler connected -> Global Logging")
if __name__ == '__main__':
app.start()
then you can define MyLogHandler() as you wish.
To send the logs to STDOUT you should also be able to use (I have not tested it):
my_handler = logging.StreamHandler(sys.stdout)
I have minimalistic Tornado application:
import tornado.ioloop
import tornado.web
class PingHandler(tornado.web.RequestHandler):
def get(self):
self.write("pong\n")
if __name__ == "__main__":
application = tornado.web.Application([ ("/ping", PingHandler), ])
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()
Tornado keeps reporting error requests to stderr:
WARNING:tornado.access:404 GET / (127.0.0.1) 0.79ms
Question: It want to prevent it from logging error messages. How?
Tornado version 3.1; Python 2.6
Its clear that "someone" is initializing logging subsystem when we start Tornado. Here is the code from ioloop.py that reveals the mystery:
def start(self):
if not logging.getLogger().handlers:
# The IOLoop catches and logs exceptions, so it's
# important that log output be visible. However, python's
# default behavior for non-root loggers (prior to python
# 3.2) is to print an unhelpful "no handlers could be
# found" message rather than the actual log entry, so we
# must explicitly configure logging if we've made it this
# far without anything.
logging.basicConfig()
basicConfig is called and configures default stderr handler.
So to setup proper logging for tonado access, you need to:
Add a handler to tornado.access logger: logging.getLogger("tornado.access").addHandler(...)
Disable propagation for the above logger: logging.getLogger("tornado.access").propagate = False. Otherwise messages will arrive BOTH to your handler AND to stderr
The previous answer was correct, but a little incomplete. This will send everything to the NullHandler:
hn = logging.NullHandler()
hn.setLevel(logging.DEBUG)
logging.getLogger("tornado.access").addHandler(hn)
logging.getLogger("tornado.access").propagate = False
You could also quite simply (in one line) do:
logging.getLogger('tornado.access').disabled = True
I know how to make twisted use python logging (docs)
But normal python logging is still swallowed. Print-statements are visible, but logger.warn('...') is not.
I use this logging set up in my library which I want to use from twisted:
import logging
logger=logging.getLogger(os.path.basename(sys.argv[0]))
class Foo:
def foo(self):
logger.warn('...')
I don't want to change my library to use twisted logging, since the library is already used in a lot of projects which don't use twisted.
If I google for this problem, I only find solutions which pipe the twisted logs in to the python logging.
How can I view see the logging of my library (without changing it)?
The thing is that Twisted is asynchronous, and avoids doing blocking I/O wherever possible. However, stdlib logging is not asynchronous, and so it does blocking I/O, and the two can't easily mix because of this. You may be able to achieve some measure of cooperation between them if you e.g. use a QueueHandler (introduced in the stdlib in Python 3.2 and mentioned here, but available to earlier versions through the logutils project). You can use this handler (and this handler only) to deal with events sent using stdlib logging, and your corresponding QueueListener can dispatch the events received using Twisted (non-blocking) I/O. It should work, as the queue handler shouldn't block if created with no finite capacity, and assuming that the I/O sinks can get rid of the events quickly enough (otherwise, memory would fill up).
utils/log.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import logging
import logging.handlers
from logging.config import dictConfig
from twisted.python.failure import Failure
from twisted.python import log as twisted_log
logger = logging.getLogger(__name__)
DEFAULT_LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'loggers': {
'twisted':{
'level': 'ERROR',
}
}
}
def failure_to_exc_info(failure):
"""Extract exc_info from Failure instances"""
if isinstance(failure, Failure):
return (failure.type, failure.value, failure.getTracebackObject())
def configure_logging(logfile_path):
"""
Initialize logging defaults for Project.
:param logfile_path: logfile used to the logfile
:type logfile_path: string
This function does:
- Assign INFO and DEBUG level to logger file handler and console handler
- Route warnings and twisted logging through Python standard logging
"""
observer = twisted_log.PythonLoggingObserver('twisted')
observer.start()
dictConfig(DEFAULT_LOGGING)
default_formatter = logging.Formatter(
"[%(asctime)s] [%(levelname)s] [%(name)s] [%(funcName)s():%(lineno)s] [PID:%(process)d TID:%(thread)d] %(message)s",
"%d/%m/%Y %H:%M:%S")
file_handler = logging.handlers.RotatingFileHandler(logfile_path, maxBytes=10485760,backupCount=300, encoding='utf-8')
file_handler.setLevel(logging.INFO)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(default_formatter)
console_handler.setFormatter(default_formatter)
logging.root.setLevel(logging.DEBUG)
logging.root.addHandler(file_handler)
logging.root.addHandler(console_handler)
hello.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import logging
logger = logging.getLogger(__name__)
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.resource import Resource
class BasicPage(Resource):
isLeaf = True
def render_GET(self, request):
logger.info("<html><body><h1>Basic Test</h1><p>This is a basic test page.</p></body></html>")
return "<html><body><h1>Basic Test</h1><p>This is a basic test page.</p></body></html>"
def hello():
logger.info("Basic web server started. Visit http://localhost:8000.")
root = BasicPage()
factory = Site(root)
reactor.listenTCP(8000, factory)
reactor.run()
exit()
main.py
def main():
configure_logging('logfilepath')
hello.hello()