I'm working with a Python/Flask application and trying to get the logs to be formatted (by line) in json.
Using the python-json-logger package, I've modified the formatter for the app.logger as follows:
from pythonjsonlogger import jsonlogger
formatter = jsonlogger.JsonFormatter(
'%(asctime) %(levelname) %(module) %(funcName) %(lineno) %(message)')
app.logger.handlers[0].setFormatter(formatter)
This works as expected. Any messages passed to app.logger are correctly formatted in json.
However the application is also automatically logging all requests. This information shows up in stdout as follows:
127.0.0.1 - - [19/Jun/2015 12:22:03] "GET /portal/ HTTP/1.1" 200 -
I want this information to be formatted in json as well. I've been looking for the logger/code that is responsible for creating this output with no success.
Where is this output generated?
Are the mechanisms to change the formatting of this logged information?
When you use the app.logger attribute for the first time, Flask sets up some log handlers:
a debug logger that is set to log level DEBUG and that filters on app.debug being true.
a production logger that is set to ERROR.
You can remove those handlers again yourself, by running:
app.logger.handlers[:] = []
However, the log line you see is not logged by Flask, but by the WSGI server. Both the built-in Werkzeug server (used when you use app.run()) and various other WSGI servers do this. Gunicorn for example uses the default Python logger to record access.
The built-in WSGI server should never be used in production (it won't scale well, and is not battle-hardened against malicious attackers).
For Gunicorn, you can disable log propagation to keep the logging separate:
logging.getLogger('gunicorn').propagate = False
Flask internally uses werkzeug server. Those logs are printed by it, not flask. You can access to werkzeug logger with logging.getLogger('werkzeug') and configure as you wish. For example:
werkzeug = logging.getLogger('werkzeug')
if len(werkzeug.handlers) == 1:
formatter = logging.Formatter('%(message)s', '%Y-%m-%d %H:%M:%S')
werkzeug.handlers[0].setFormatter(formatter)
Related
This is the configuration I have right now
logging.basicConfig(format='%(levelname)s: %(message)s', level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.addHandler(AzureLogHandler(
connection_string='InstrumentationKey='+app_insights_id)
)
#app.route('/', methods = ['GET','POST'])
def main_page():
logger.debug('Hello Flask, on Azure App Service for Linux!') <--- want this to be printed on console
return 'Hello Flask, on Azure App Service for Linux!'
I am pretty sure this is possible since it was working for me until a couple of days back and after some change I cannot trace back, all logs stopped printing to console
I have tried setting the environment variable FLASK_ENV to development. That didn't work either.
According to this answer, we need to get logger from werkzeug, is there any way apart from this to get logs in console after starting flask?
The fastAPI library that I import for an API I have written, writes many logging.INFO level messages to the console, which I would like either to redirect to a file-based log, or ideally, to both console and file. Here is an example of fastAPI module logging events in my console:
So I've tried to implement this Stack Overflow answer ("Easy-peasy with Python 3.3 and above"), but the log file it creates ("api_screen.log") is always empty....
# -------------------------- logging ----------------------------
logging_file = "api_screen.log"
logging_level = logging.INFO
logging_format = ' %(message)s'
logging_handlers = [logging.FileHandler(logging_file), logging.StreamHandler()]
logging.basicConfig(level = logging_level, format = logging_format, handlers = logging_handlers)
logging.info("------logging test------")
Even though my own "------logging test------" message does appear on console within the other fastAPI logs:
As you can see here it's created the file, but it has size zero.
So what do I need to do also to get the file logging working?
There are multiple issues here. First and most importantly: basicConfig does nothing if a logger is already configured, which fastAPI does. So the handlers you are creating are never used. When you call logging.info() you are sending a log to the root logger which is printed because the fastAPI has added a handler to it. You are also not setting the level on your handlers. Try this code instead of what you currently have:
logging_file = "api_screen.log"
logging_level = logging.INFO
logging_fh = logging.FileHandler(logging_file)
logging_sh = logging.StreamHandler()
logging_fh.setLevel(logging_level)
logging_sh.setLevel(logging_level)
root_logger = logging.getLogger()
root_logger.addHandler(logging_fh)
root_logger.addHandler(logging_sh)
logging.info('--test--')
I'm using python's logging module. I've initialized it as:
import logging
logger = logging.getLogger(__name__)
in every of my modules. Then, in the main file:
logging.basicConfig(level=logging.INFO,filename="log.txt")
Now, in the app I'm also using WSGIServer from gevent. The initializer takes a log argument where I can add a logger instance. Since this is an HTTP Server it's very verbose.
I would like to log all of my app's regular logs to "log.txt" and WSGIServer's logs to "http-log.txt".
I tried this:
logging.basicConfig(level=logging.INFO,filename="log.txt")
logger = logging.getLogger(__name__)
httpLogger = logging.getLogger("HTTP")
httpLogger.addHandler(logging.FileHandler("http-log.txt"))
httpLogger.addFilter(logging.Filter("HTTP"))
http_server = WSGIServer(('0.0.0.0', int(config['ApiPort'])), app, log=httpLogger)
This logs all HTTP messages into http-log.txt, but also to the main logger.
How can I send all but HTTP messages to the default logger (log.txt), and HTTP messages only to http-log.txt?
EDIT: Since people are quickly jumping to point that this Logging to two files with different settings has an answer, plese read the linked answer and you'll see they don't use basicConfig but rather initialize each logger separately. This is not how I'm using the logging module.
Add the following line to disable propagation:
httpLogger.propagate = False
Then, it will no longer propagate messages to its ancestors' handlers which includes the root logger for which you have set up the general log file.
I'm trying to see the difference between default flask logger and stackdriver logger in GAE's sample application: https://cloud.google.com/python/getting-started/using-pub-sub
Code without StackDriver logger:
def create_app(config, debug=False, testing=False, config_overrides=None):
app = Flask(__name__)
app.config.from_object(config)
app.debug = debug
app.testing = testing
if config_overrides:
app.config.update(config_overrides)
# Configure logging
if not app.testing:
logging.basicConfig(level=logging.INFO)
Code with StackDriver logger:
def create_app(config, debug=False, testing=False, config_overrides=None):
app = Flask(__name__)
app.config.from_object(config)
app.debug = debug
app.testing = testing
if config_overrides:
app.config.update(config_overrides)
# [START setup_logging]
if not app.testing:
client = google.cloud.logging.Client(app.config['PROJECT_ID'])
# Attaches a Google Stackdriver logging handler to the root logger
client.setup_logging(logging.INFO)
There's some difference with the StackDriver code where a logger was imported from google cloud. However, the output of the logs seems similar:
Output Log without StackDriver:
Output Log with StackDriver:
These logs does not look that different with or without a StackDriver.
When I go to the StackDriver logs I get redirected to the default logs in GAE. Is there anything special with StackDriver loggers that the normal flask logger cannot do?
Taking a look into the the two function that you are using for logger configuration: Basicconfig and Setup.logging, your loggers have similar settings so it make sense to me to have similar log output.
I didn't understand what you expected to see in Stackdriver Logging Viewer since the two pictures that you attached looks right to me since them are normal Log entry for Stackdriver Logging. Notice that, by default, App Engine logging is provided by Stackdriver Logging as explained in this document
The advantage of Stackdriver Logging is to have a better management of the logs and the possibility to analyze them. You can have a look in this tutorial in order to have an idea about it.
When debugging level of main logger in Pyramid app is set to DEBUG, transaction is spewing lots of pointless debug messages.
In Nosetests I can disable that this way:
from transaction._compat import get_thread_ident
txn_logger = logging.getLogger("txn.%d" % get_thread_ident())
txn_logger.setLevel(logging.WARN)
However, in Pyramid app the infrastructure adds "scoped session" to each HTTP request and that obviously means get_thread_ident() is different every time.
Is there some way of disabling that globally without repeating above in every single Pyramid view?
Simply turn off logging for the txn parent logger in your logging config.
[loggers]
keys = transactions, ...
[logger_transactions]
level = WARN
handlers =
qualname = txn