I know how to make twisted use python logging (docs)
But normal python logging is still swallowed. Print-statements are visible, but logger.warn('...') is not.
I use this logging set up in my library which I want to use from twisted:
import logging
logger=logging.getLogger(os.path.basename(sys.argv[0]))
class Foo:
def foo(self):
logger.warn('...')
I don't want to change my library to use twisted logging, since the library is already used in a lot of projects which don't use twisted.
If I google for this problem, I only find solutions which pipe the twisted logs in to the python logging.
How can I view see the logging of my library (without changing it)?
The thing is that Twisted is asynchronous, and avoids doing blocking I/O wherever possible. However, stdlib logging is not asynchronous, and so it does blocking I/O, and the two can't easily mix because of this. You may be able to achieve some measure of cooperation between them if you e.g. use a QueueHandler (introduced in the stdlib in Python 3.2 and mentioned here, but available to earlier versions through the logutils project). You can use this handler (and this handler only) to deal with events sent using stdlib logging, and your corresponding QueueListener can dispatch the events received using Twisted (non-blocking) I/O. It should work, as the queue handler shouldn't block if created with no finite capacity, and assuming that the I/O sinks can get rid of the events quickly enough (otherwise, memory would fill up).
utils/log.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import logging
import logging.handlers
from logging.config import dictConfig
from twisted.python.failure import Failure
from twisted.python import log as twisted_log
logger = logging.getLogger(__name__)
DEFAULT_LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'loggers': {
'twisted':{
'level': 'ERROR',
}
}
}
def failure_to_exc_info(failure):
"""Extract exc_info from Failure instances"""
if isinstance(failure, Failure):
return (failure.type, failure.value, failure.getTracebackObject())
def configure_logging(logfile_path):
"""
Initialize logging defaults for Project.
:param logfile_path: logfile used to the logfile
:type logfile_path: string
This function does:
- Assign INFO and DEBUG level to logger file handler and console handler
- Route warnings and twisted logging through Python standard logging
"""
observer = twisted_log.PythonLoggingObserver('twisted')
observer.start()
dictConfig(DEFAULT_LOGGING)
default_formatter = logging.Formatter(
"[%(asctime)s] [%(levelname)s] [%(name)s] [%(funcName)s():%(lineno)s] [PID:%(process)d TID:%(thread)d] %(message)s",
"%d/%m/%Y %H:%M:%S")
file_handler = logging.handlers.RotatingFileHandler(logfile_path, maxBytes=10485760,backupCount=300, encoding='utf-8')
file_handler.setLevel(logging.INFO)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(default_formatter)
console_handler.setFormatter(default_formatter)
logging.root.setLevel(logging.DEBUG)
logging.root.addHandler(file_handler)
logging.root.addHandler(console_handler)
hello.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import logging
logger = logging.getLogger(__name__)
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.resource import Resource
class BasicPage(Resource):
isLeaf = True
def render_GET(self, request):
logger.info("<html><body><h1>Basic Test</h1><p>This is a basic test page.</p></body></html>")
return "<html><body><h1>Basic Test</h1><p>This is a basic test page.</p></body></html>"
def hello():
logger.info("Basic web server started. Visit http://localhost:8000.")
root = BasicPage()
factory = Site(root)
reactor.listenTCP(8000, factory)
reactor.run()
exit()
main.py
def main():
configure_logging('logfilepath')
hello.hello()
Related
I want to use the logging module, but I'm having some trouble because it is outputting twice. I've read a lot of posts with people having the same issue and log.propagate = False or `log.handlers.pop()´ fixed it for them. This doesn't work for me.
I have a file called logger.py that looks like this:
import logging
def __init__():
log = logging.getLogger("output")
filelog = logging.FileHandler("output.log")
formatlog = logging.Formatter("%(asctime)s %(levelname)s %(message)s")
filelog.setFormatter(formatlog)
log.addHandler(filelog)
log.setLevel(logging.INFO)
return log
So that I from multiple files can write:
import logger
log = logger.__init__()
But this is giving me issues. So I've seen several solutions, but I don't know how to incorporate it in multiple scripts without defining the logger in all of them.
I found a solution which was really simple. All I needed to do was to add an if statement checking if the handlers already existed. So my logger.py file now looks like this:
import logging
def __init__():
log = logging.getLogger("output")
if not log.handlers:
filelog = logging.FileHandler("output.log")
formatlog = logging.Formatter("%(asctime)s %(levelname)s %(message)s")
filelog.setFormatter(formatlog)
log.addHandler(filelog)
log.setLevel(logging.INFO)
return log
multiple scripts lead to multiple processes. so you unluckily create multiple objects returned from the logger.__init__() function.
usually, you have one script which creates the logger and the different processes,
but as i understand, you like to have multiple scripts logging to the same destination.
if you like to have multiple processes to log to the same destination, i recommend using inter process communication as "named pipes" or otherwise using a UDP/TCP port for logging.
There are also queue modules available in python to send a (atomic) logging entry to be logged in one part (compared to append to a file by multiple processes - which may lead from 111111\n and 22222\n to 11212121221\n\n in the file)
otherwise think about named pipes...
Code snippet for logging server
Note: i just assumed to log everything as error...
import socket
import logging
class mylogger():
def __init__(self, port=5005):
log = logging.getLogger("output")
filelog = logging.FileHandler("output.log")
formatlog = logging.Formatter("%(asctime)s %(levelname)s %(message)s")
filelog.setFormatter(formatlog)
log.addHandler(filelog)
log.setLevel(logging.INFO)
self.log = log
UDP_IP = "127.0.0.1" # localhost
self.port = port
self.UDPClientSocket = socket.socket(family=socket.AF_INET, type=socket.SOCK_DGRAM)
self.UDPClientSocket.bind((UDP_IP, self.port))
def pollReceive(self):
data, addr = self.UDPClientSocket.recvfrom(1024) # buffer size is 1024 bytes
print("received message:", data)
self.log.error( data )
pass
log = mylogger()
while True:
log.pollReceive()
You have to be very carefully by adding new handler to your logger with:
log.addHandler(...)
If you add more than one handler to your logger, you will get more than one outputs. Be aware that this is only truth for logger which are running in the same thread. If you are running a class which is derived from Thread, then it is the same thread. But if you are running a class which is derived from Process, then it is another thread. To ensure that you only have one handler for your logger according to this thread, you should use the following code (this is an example for SocketHandler):
logger_root = logging.getLogger()
if not logger_root.hasHandlers():
socket_handler = logging.handlers.SocketHandler('localhost', logging.handlers.DEFAULT_TCP_LOGGING_PORT)
logger_root.addHandler(socket_handler)
So Celery is a super great library but its documentation for the logging section isn't the best, which bring me here to request for assistance.
My Script as of now is as so (well in summary):
import logging
from celery import Celery
from celery.utils.log import get_logger
from task import process
import config
logger = get_logger(__name__)
timber_handler = timber.TimberHandler(api_key=config.key,
level=logging.INFO)
logger.addHandler(timber_handler)
app = Celery('task',
broker=config.url,
backend='rpc://')
#app.task
def run_task():
status = get_status() # get alive or dead status
if status == 1:
logger.info("Task is running")
process()
#app.on_after_configure.connect
def task_periodic(**kwargs):
app.add_periodic_task(2.0, run_task.s(), name="Run Constantly")
# More tasks
The process function in the tasks.py file is very basic function hitting up APIs and DBs for some info and I want to be able to log that to a logger (timber.io) which attaches to the python logging library and is an online storage for logs.
However, my major issue is that the logs are getting sent to stdout and not to the timber logs. I have looked at celery.signals but the documentation isn't great. Any assistance here would be greatly appreciated. Thank you.
Can you try this?
import logging
import os
import sys
from celery import Celery
from celery.signals import after_setup_logger
app = Celery('app')
app.conf.update({
'broker_url': 'filesystem://',
'broker_transport_options': {
'data_folder_in': './broker/out',
'data_folder_out': './broker/out',
'data_folder_processed': './broker/processed'
},
'result_persistent': False,
'task_serializer': 'json',
'result_serializer': 'json',
'accept_content': ['json']})
logger = logging.getLogger(__name__)
for f in ['./broker/out', './broker/processed']:
if not os.path.exists(f):
os.makedirs(f)
#after_setup_logger.connect
def setup_loggers(logger, *args, **kwargs):
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# add filehandler
fh = logging.FileHandler('logs.log')
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
logger.addHandler(fh)
#app.task()
def add(x, y):
logger.info('Found addition')
logger.info('Added {0} and {1} to result, '.format(x,y))
return x+y
if __name__ == '__main__':
task = add.s(x=2, y=3).delay()
Start the worker like this:
celery worker --app=app.app --concurrency=1 --loglevel=INFO
And kick off the task asynchronously:
python app.py
I've changed it so it's a stand-alone script that just uses the filesystem as a message broker (also, I've deliberately replaced the timber.io handler with a filehandler).
This writes the logs to logs.log (replace the filehandler with the timber.io handler and that should solve your issue).
I had a bit of a hard time as the I couldn't get it working with worker_hijack_root_logger=False and a custom logger defined in setup_logging.
However, after revisiting the docs, I came to the conclusion it's a better option to not override the logger but just augment it:
If you’d like to augment the logging configuration setup by Celery
then you can use the after_setup_logger and after_setup_task_logger
signals.
See also: http://docs.celeryproject.org/en/latest/userguide/signals.html#after-setup-logger
I am using wsgi server to spawn the servers for my web application. I am having problem with information logging.
This is how I am running the app
from gevent import monkey; monkey.patch_all()
from logging.handlers import RotatingFileHandler
import logging
from app import app # this imports app
# create a file to store weblogs
log = open(ERROR_LOG_FILE, 'w'); log.seek(0); log.truncate();
log.write("Web Application Log\n"); log.close();
log_handler = RotatingFileHandler(ERROR_LOG_FILE, maxBytes =1000000, backupCount=1)
formatter = logging.Formatter(
"[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s"
)
log_handler.setFormatter(formatter)
app.logger.setLevel(logging.DEBUG)
app.logger.addHandler(log_handler)
# run the application
server= wsgi.WSGIServer(('0.0.0.0', 8080), app)
server.serve_forever()
However, on running the application it is not logging anything. I guess it must be because of WSGI server because app.logger works in the absence of WSGI. How can I log information when using WSGI?
According to the gevent uwsgi documentation you need to pass your log handler object to the WSGIServer object at creation:
log – If given, an object with a write method to which request (access) logs will be written. If not given, defaults to sys.stderr. You may pass None to disable request logging. You may use a wrapper, around e.g., logging, to support objects that don’t implement a write method. (If you pass a Logger instance, or in general something that provides a log method but not a write method, such a wrapper will automatically be created and it will be logged to at the INFO level.)
error_log – If given, a file-like object with write, writelines and flush methods to which error logs will be written. If not given, defaults to sys.stderr. You may pass None to disable error logging (not recommended). You may use a wrapper, around e.g., logging, to support objects that don’t implement the proper methods. This parameter will become the value for wsgi.errors in the WSGI environment (if not already set). (As with log, wrappers for Logger instances and the like will be created automatically and logged to at the ERROR level.)
so you should be able to do wsgi.WSGIServer(('0.0.0.0', 8080), app, log=app.logger)
You can log like this -
import logging
import logging.handlers as handlers
.
.
.
logger = logging.getLogger('MainProgram')
logger.setLevel(10)
logHandler = handlers.RotatingFileHandler(filename.log,maxBytes =1000000, backupCount=1)
logger.addHandler(logHandler)
logger.info("Logging configuration done")
.
.
.
# run the application
server= wsgi.WSGIServer(('0.0.0.0', 8080), app, log=logger)
server.serve_forever()
I'm trying to use standard Python logging module alongside with gevent. I have monkey patched threading and I expect logging to work with my app:
import gevent.monkey
gevent.monkey.patch_all()
import logging
logger = logging.getLogger()
fh = logging.FileHandler('E:\\spam.log')
fh.setLevel(logging.DEBUG)
logger.addHandler(fh)
def foo():
logger.debug('Running in foo')
gevent.sleep(0)
logger.debug('Explicit context switch to foo again')
def bar():
logger.debug('Explicit context to bar')
gevent.sleep(0)
logger.debug('Implicit context switch back to bar')
gevent.joinall([
gevent.spawn(foo),
gevent.spawn(bar),
])
Unfortunately E:\spam.log is empty and I can see no output to the console. It seems that either I haven't configured logging properly or gevent does not support it at all (which I don't believe, because gevent documentation says that it does). So, how can I log at gevent app?
You haven't configured it correctly. You need to set the DEBUG level on the logger rather than the handler, otherwise the logger's default level (WARNING) causes your debug messages to be dropped. Try doing a
logger.setLevel(logging.DEBUG)
and it should work.
I use logging to one file with different scripts of one program - such as cron tasks, twisted daemons (HttpServers with some data) etc.
If I use default Python logging in base class such as
import logging
import logging.handlers
....
self.__fname = open(logname, 'a')
logging.basicConfig(format=FORMAT, filename=logname, handler=logging.handlers.RotatingFileHandler)
self._log = logging.getLogger(self.__pname)
self._log.setLevel(loglevel)
logging.warn('%s %s \033[0m' % (self.__colors[colortype], msg))
then it's work normally, sending output of all scripts in one file, but some important part of default twisted log missing - such as info about http request/headers etc
else I use twisted logging such as
from twisted.python.logfile import DailyLogFile
from twisted.python import log
from twisted.application.service import Application
....
application = Application("foo")
log.startLogging(DailyLogFile.fromFullPath(logname))
print '%s %s \033[0m' % (self.__colors[colortype], msg)
then works with additional data, but some trouble with logging from different scripts exists - looks like cron tasks trouble appears. Looks like these cron tasks switch context of output and some part of logging output is missing and not restored
Of, course - cron tasks working without Twisted reactor, but using twisted logging.
What I should do with logging for log all data printed both Twisted/cron parts of app?
Thanks for any help!
I think the point is that you should not use DailyLogFile but use PythonLOggingObserver to redirect the log to standard lib log
from twisted.python import log
observer = log.PythonLoggingObserver()
observer.start()
log.msg('%s %s \033[0m' % (self.__colors[colortype], msg))
Also you might want to see the example in docs: http://twistedmatrix.com/documents/current/core/howto/logging.html#auto3