I have written a python module that I'm using in my django app, so that it can be generalised. I have a django mangement function that calls this library.
Inside the library I've tried to use proper python logging. I have written a simple command line programme that calls the library and dumps logging data to stdout, and that works. However when this library is called from the django management command, there is logging output from the django management command, however there is no logging output from the library. It is as if there is no logging calls in the library.
I would like the logging output from the library and the django management command to appear on stdout. As if I just used print in both the django management command and the library.
Inside the django management command, I call this, so that all logging output goes to the terminal. I want logging output from this django management command and the library to go to the terminal.
import logging
logger = logging.getLogger(__name__)
class Command(BaseCommand):
def handle(self, *args, **options):
# ...
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
logger.addHandler(ch)
logger.setLevel(logging.DEBUG)
Inside the library, I do logger = logging.getLogger(__name__)"at the root of the file as well.
I am new using the python logging framework (usually I just print ;)) So I might be making a simple mistake or asking for something that's delibrately impossible. This is Django 1.6.5, and python 2.7.6. My python library is pure python and is single threaded, it's all very simple.
Related
I just discovered this very strange behaviour of the logging module in Spyder:
import logging
logging.getLogger("hurricane").handlers
Out[2]: [] # expected
logging.getLogger("tornado").handlers
Out[3]: [<StreamHandler <stderr> (NOTSET)>] # Where does that StreamHandler come from?!
Note that these are the first lines from a freshly started interpreter. So I haven't imported tornado or any other package except logging. Yet, unlike any other logger I tried to get, it comes with a StreamHandler attached.
Why?
Related question: How to prevent Python Tornado from logging to stdout/console?
I think Tornado uses logging for its normal-operation request logging. This is not a great idea - an application is supposed to work exactly as before if all logging is disabled (aside from the logging part, of course) but numerous web server authors use logging functionality as a convenience to do part of their normal operation rather than just using it for diagnostics. I believe you can turn this off using a logging=none configuration option; see this page for more information.
I'm typically using the standard logging library or structlog. Recently, when I tried to add a test to a function, I've seen that it uses current_app.logger - which means that this function needed an application context. Otherwise, it didn't need the application context.
Is there an advantage of using the current_app.logger instead of logger = logging.getLogger(__name__)? Is there maybe data available that is not available to the standard logger?
MVCE
import logging
from flask import Flask, current_app
app = Flask(__name__)
# Normal Python logging
logger = logging.getLogger(__name__)
ch = logging.StreamHandler()
logger.addHandler(ch)
logger.setLevel(logging.DEBUG)
# Flasks app logger
app.logger.setLevel(logging.DEBUG)
#app.route("/")
def index():
current_app.logger.info("flask-app logger info-msg")
logger.info("base-logger infomsg")
return "foo"
if __name__ == "__main__":
app.run()
Mostly it's a matter of configuration. From the Flask source code on logging:
def create_logger(app):
"""Get the Flask app's logger and configure it if needed.
The logger name will be the same as
:attr:`app.import_name <flask.Flask.name>`.
When :attr:`~flask.Flask.debug` is enabled, set the logger level to
:data:`logging.DEBUG` if it is not set.
If there is no handler for the logger's effective level, add a
:class:`~logging.StreamHandler` for
:func:`~flask.logging.wsgi_errors_stream` with a basic format.
"""
So if you use Flask's logger object, you get one configured with the log level already set based on Flask's other settings, along with a handler.
Notice that within Flask's source code, the logger is used:
def log_exception(self, exc_info):
"""Logs an exception. This is called by :meth:`handle_exception`
if debugging is disabled and right before the handler is called.
The default implementation logs the exception as error on the
:attr:`logger`.
.. versionadded:: 0.8
"""
self.logger.error(
f"Exception on {request.path} [{request.method}]", exc_info=exc_info
)
So given that Flask and presumably apps written using Flask use the current_app.logger log, it would generally be prudent to follow that pattern, since any additional configuration will also be propagated. If you had a compelling reason to, however, you could certainly use your own logger.
Setting logger = logging.getLogger(__name__) configures your root logger by default, which you might not want in a production environment. For instance when you define a logger per file/module then you might fall into a trap like the one explained here. So as explained and commented by #noslenkwah.
Having an internal logger is not unique to Flask, it's the standard way to use logging in Python. Every module defines it's own logger, but the configuration of the logging is only handled by the last link in the chain: your project.
My point - if you have a small application, you can obviously stick to a default logger. For more control over the application and setting more than one logger, you might want to check Python Comprehensive Logging using YAML Configuration and a use-case which I've tried to build from this YAML file, to integrate it in one of my hobby projects.
So I've tried to add some logger to my discord bot, to see logs in file not just in console, cause obviously it's irritating when I reset app and find out that I have to check logs that I've already destroyed, I set it up like this:
logging.basicConfig(filename='CoronaLog.log', level=logging.DEBUG, format='%(levelname)s %(asctime)s %(message)s')
And learned the hard way that discord.py library has its own logger installed so now my logs look like one big mess, is there any way to disable discord.py's logging, or at least output it to another file?
EDIT: I've tried creating two loggers, so it would look like this:
logging.basicConfig(filename='discord.log', level=logging.DEBUG, format='%(levelname)s %(asctime)s %(message)s')
nonDiscordLog = logging.getLogger('discord')
handler = logging.FileHandler(filename='CoronaLog.log', encoding='utf-8', mode='w')
handler.setFormatter(logging.Formatter('%(levelname)s %(asctime)s:%(name)s: %(message)s'))
nonDiscordLog.addHandler(handler)
So the discord log, would be logged as the basic config says to discord.log file and, when executed like this:
nonDiscordLog.info("execution took %s seconds \n" % (time.time() - startTime))
would log into CoronaLog.log file, although it didn't really change anything
discord.py is in this regard terribly unintuitive for everyone but beginners, anyway after confronting the docs you can find out that this behavior can be avoided with:
import discord
client = discord.Client(intents=discord.Intents.all())
# Your regular code here
client.run(__YOURTOKEN__, log_handler=None)
Of course, you can supplement your own logger instead of None but to answer your question exactly, this is how you can disable discord.py's logging.
There's actually a bit more that you can do with the default logging, and you can read all about it on
the official docs
https://discordpy.readthedocs.io/en/latest/logging.html says:
"discord.py logs errors and debug information via the logging python module. It is strongly recommended that the logging module is configured, as no errors or warnings will be output if it is not set up. Configuration of the logging module can be as simple as:
import logging
logging.basicConfig(level=logging.INFO)
Placed at the start of the application. This will output the logs from discord as well as other libraries that use the logging module directly to the console."
Maybe try configuring logging a different way? Because when starting logging, it appears to initialize discord.py's llogging. maybe try
import logging
# Setup logging...
import discord
Maybe if you import it afterwords, it won't set it up.
for my perspective, I would initialize a logger to log message for my app
import logging
logger = logging.getLogger('my_app')
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter(
'%(asctime)s %(levelname)s %(name)s %(message)s'))
logger.addHandler(handler)
when errors happend:
try:
blah()
except Exceptions as e:
logger.warning(e)
But I'm using some third-party module like sqlalchemy, sqlalchemy may log warning infos when error happends(e.g, varchar is too long and being truncated) and it use a separate logger (so does some other modules, like requests)
sqlalchemy/log.py
This may leads some bad issues and it's not trackable.
In case I'm using lots of third-party modules, how can I log all third-party message to a separate file to help me doing trouble shooting works?
You can set up a handler to log to a file and attach that to the root logger, but that will log all messages, even ones from your code. To leave your messages out of that file, use a logging.Filter subclass which filters out all messages which aren't from one of your code's top-level package namespaces. Attach that filter to your handler.
My cherrypy application (3.2.2 on Python 2.6) is using 3rd party libs, and these libs use standard logging internally, like so:
logger = logging.getLogger(__name__)
logger.info("a message from some library")
Now, in my cherrypy config, I have:
log.access_file = '/path/access.log'
log.error_file = '/path/error.log'
but only CP messages ever appear in these two files, not any of the other logging. But I need all logging there, not just the logging that CP itself issues internally.
Is there a way to capture all output, including the 3rd party logging (which I assume goes into stdout/stderr and then disappears, as the process is a detached daemon), into /path/error.log? Plug arbitrary stdout/stderr into CP's log somehow?
Is this what you're looking for? All output into the error log?
python yourCherryServer.py &>> /path/error.log