I have a basic logger set up using the logging library in Python 3.10.4. I'm attempting to make the FastAPI exception handler log any exceptions it handles, to no avail. The handler is running, as I can see that my custom Response is returned, but no log.
logging.basicConfig(filename='mylan.log')
logger = logging.getLogger("MyLAN")
logger.setLevel(logging.DEBUG)
discord_handler = DiscordHandler(webhook_url, agent, notify_users=notify_users, emit_as_code_block=False, max_size=2000)
# Add log level to handlers
discord_handler.setLevel(logging.DEBUG)
# Add format to handlers
discord_handler.setFormatter(FORMAT)
# Add the handlers to the Logger
logger.addHandler(discord_handler)
logger.debug("Logger created")
app = FastAPI()
logger.debug('test') # This works
#app.exception_handler(Exception)
def handle_exception(req, exc):
logger.debug("Something's brokey") # This does not
return Response("Internal Server Error Test", status_code=500)
I can also confirm that the logger works, as it logs a simple message on startup which is saved successfully.
I'm not even getting any errors in stdout that might guide me towards a solution.
Anyone have any ideas?
It works for me as written. Here's the code I tested. It's basically identical to what you have, but it's runnable, and includes a /error route that raises a KeyError to test the exception handler:
import logging
from fastapi import FastAPI, Response
from discord_handler import DiscordHandler
webhook_url = "https://discord.com/api/webhooks/..."
logging.basicConfig(filename="mylan.log")
logger = logging.getLogger("MyLAN")
logger.setLevel(logging.DEBUG)
discord_handler = DiscordHandler(
webhook_url,
"Example",
notify_users=[],
emit_as_code_block=True,
max_size=2000,
)
# Add log level to handlers
discord_handler.setLevel(logging.DEBUG)
# Add the handlers to the Logger
logger.addHandler(discord_handler)
logger.debug("Logger created")
app = FastAPI()
logger.debug("test") # This works
#app.exception_handler(Exception)
def handle_exception(req, exc):
logger.debug("Something's broken") # This does not
return Response("Internal Server Error Test", status_code=500)
#app.get("/error")
def error():
raise KeyError("example")
When this code starts up, I see in my discord channel:
Logger created...
test...
And when I make a request to /error, I see:
Something's broken...
Related
I have a logger defined as in logger.py:
def my_logger(module_name):
logger = logging.getLogger(module_name)
logger.setLevel(logging.DEBUG)
# stream handler
# debug handler
f1_handler = logging.handlers.RotatingFileHandler(os.path.join(abspath, "logs/debug.log"))
f1_handler.setLevel(logging.DEBUG)
f1_handler.setFormatter(format)
# warning handler
f3_handler = logging.handlers.RotatingFileHandler(os.path.join(abspath, "logs/warning.log"))
f3_handler.setLevel(logging.WARNING)
f3_handler.setFormatter(format)
# error handler
f4_handler = logging.handlers.RotatingFileHandler(os.path.join(abspath, "logs/error.log"))
f4_handler.setLevel(logging.ERROR)
f4_handler.setFormatter(format)
# Add handlers to the logger
logger.addHandler(f1_handler)
logger.addHandler(f3_handler)
logger.addHandler(f4_handler)
return logger
Then in my test.py, I can test the logger as test.py:
from kg import logger
logger = logger.my_logger(__name__)
if __name__ == "__main__":
logger.info('Start main ...')
This works as expected, the logger message only goes to debug.log file under my logs/debug.log, meaning that it doesn't print the message to console.
However, if I test it in a Flask application, and in browser I sent a request, it always print the log message to console as well, in addition to the log file.
My flask api in flask_test.py:
from search import keyword_match
#api.route('/match', methods=['GET'])
def match():
text, parameters = _parse_parameters()
json_result = keyword_match(text, **parameters)
return json_result
if __name__ == '__main__':
#api.run(host='0.0.0.0', port='5000')
logger.info('Test log in Flask api!')
from waitress import serve
serve(api, host="0.0.0.0", port=5000)
The logger here in the 'main' still works as expected, it doesn't print out to console, since no stream handler is configured in my logger. However, there is a logger message in search.py, which defines the keyword_match function:
from kg import logger
logger = logger.my_logger(__name__)
def keyword_match(keyword, **args):
logger.info('keyword: {}'.format(keyword))
...
It is in this file, it always prints out the log info onto console even if there is no stream handler in my own logger definition. It seems Flask stream handler is always turned on and how to turn it off? I tried this in my flask_test.py, but it didn't helper:
logger = logger.my_logger(__name__)
app = Flask(__name__)
from flask.logging import default_handler
app.logger.removeHandler(default_handler)
What's the problem?
Have you tried moving/removing this line of code: logger = logger.my_logger(__name__) to after you remove the default handler?
From my understanding, flask has a default understanding of what the logger object is. But having that line of code before you reference app.logger means that the meaning of logger has been changed.
app = Flask(__name__)
from flask.logging import default_handler
app.logger.removeHandler(default_handler)
logger = logger.my_logger(__name__)
I want to make log files when I run my minimal flask app, basically, I want to log all logging messages from the python console when different endpoints are used. To do so, I used a flask logging mechanism but couldn't catch logging messages including a running sessions on python concole, logging info from testing different endpoints with the specific routes. How can I catch running session info from python console? How do I make log for errors, warning, session info from python console? Is there any way to make this happen? any thought?
my current attempt
here is my current attempt which failed to catch log info on python console:
import os, logging, logging.handlers, traceback
from flask import Flask, jsonify, request
from flask_restplus import Api, Resource, Namespce,fields, reqparse, inputs
def primes_method1(n):
out = list()
for num in range(1, n+1):
prime = True
for i in range(2, num):
if (num % i == 0):
prime = False
if prime:
out.append(num)
return out
def getLogger(logname, logdir, logsize=500*1024, logbackup_count=4):
if not os.path.exists(logdir):
os.makedirs(logdir)
logfile='%s/%s.log' % (logdir, logname)
loglevel = logging.INFO
logger = logging.getLogger(logname)
logger.setLevel(loglevel)
if logger.handlers is not None and len(logger.handlers) >= 0:
for handler in logger.handlers:
logger.removeHandler(handler)
logger.handlers = []
loghandler = logging.handlers.RotatingFileHandler(
logfile, maxBytes=logsize, backupCount=logbackup_count)
formatter = logging.Formatter('%(asctime)s-%(name)s-%(levelname)s-%(message)s')
loghandler.setFormatter(formatter)
logger.addHandler(loghandler)
return logger
app = Flask(__name__)
api = Api(app)
ns = api.namespace('ns')
payload = api.model('Payload', {
'num': fields.Integer(required=True)
})
logger = getLogger('testLog', '~/')
#ns.route('/primenumber')
class PrimeResource(Resource):
#ns.expect(payload)
def post(self):
logger.info("get prime number")
param = request.json['num']
try:
res = primes_method1(param)
return jsonify({'output': res})
except:
return None, 400
ns1 = Namespce('')
#ns1.route('/log')
class logResource(Resource):
def get(self):
logger.info("return saved all logers from above endpoints")
if __name__ == '__main__':
api.add_namespace(ns)
api.add_namespace(ns1)
app.run(debug=True)
basically, when I test endpoint with sample data, I want to see all logged messages at #ns1.route('/log') endpoint. In my attempt, I couldn't catch running session info on python console. How to log flask running session info on python console? Is there any way to do this?
Your question isn't really related to Flask, unless I'm overlooking something unique to Flask that doesn't conform to the logging interface. This isn't a workaround either, just normal usage of the tools.
Try to add a handler to your logger like so:
loghandler = logging.handlers.RotatingFileHandler(
logfile, maxBytes=logsize, backupCount=logbackup_count)
formatter = logging.Formatter('%(asctime)s-%(name)s-%(levelname)s-%(message)s')
loghandler.setFormatter(formatter)
logger.addHandler(loghandler)
loghandler2 = logging.StreamHandler()
loghandler2.setFormatter(formatter)
logger.addHandler(loghandler2)
return logger
I want to create a custom python logging Handler to send messages via slack.
I found this package however its no longer being updated so i created a very bare bone version of it. however it doesnt seem to work, I added a print call for debugging purposes and emit is not being evoked.
import logging
# import slacker
class SlackerLogHandler(logging.Handler):
def __init__(self, api_key, channel):
super().__init__()
self.channel = channel
# self.slacker = slacker.Slacker(api_key)
def emit(self, record):
message = self.format(record)
print('works')
# self.slacker.chat.post_message(text=message, channel=self.channel)
slack_handler = SlackerLogHandler('token', 'channel')
slack_handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
slack_handler.setFormatter(formatter)
logger = logging.getLogger('my_app')
logger.addHandler(slack_handler)
logger.info('info_try') # 'works' is not printed and no slack message is sent
I saw this answer and tried to also inherit from StreamHandler but to no avail.
I think I am missing something very basic.
edited to remove slack logic for ease of reproduction.
After some additional searching I found out that the logging level that is set for the handler is separate from the level that is set for the logger.
meaning, that adding:
logger.setLevel(logging.INFO)
fixes the problem.
I have the following script that I want only the "DEBUG" log messages to be logged to the file, and nothing to the screen.
from flask import Flask, request, jsonify
from gevent.pywsgi import WSGIServer
import usaddress
app = Flask(__name__)
from logging.handlers import RotatingFileHandler
import logging
#logging.basicConfig(filename='error.log',level=logging.DEBUG)
# create a file to store weblogs
log = open('error.log', 'w'); log.seek(0); log.truncate();
log.write("Web Application Log\n"); log.close();
log_handler = RotatingFileHandler('error.log', maxBytes =1000000, backupCount=1)
formatter = logging.Formatter(
"[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s"
)
log_handler.setFormatter(formatter)
app.logger.setLevel(logging.DEBUG)
app.logger.addHandler(log_handler)
#app.route('/')
def hello():
return "Hello World!"
#app.route("/parseAddress", methods=["POST"])
def parseAddress():
address = request.form['address']
return jsonify(usaddress.tag(address)), 200
if __name__ == '__main__':
# app.run(host='0.0.0.0')
http_server = WSGIServer(('', 5000), app, log=app.logger)
http_server.serve_forever()
But right now even "INFO" messages are being logged to the file and to the screen. How can I have only the "DEBUG" messages logged to the file and nothing to the screen?
I did some interactive testing and it looks like app.logger is a logger named with python file
print(app.logger.name) # filename
and it has one handler
print(app.logger.handlers) # [<StreamHandler <stderr> (NOTSET)>]
A handler with level logging.NOTSET processes all messages (from all logging levels). So when you set app.logger.setLevel(logging.DEBUG) then all debug and higher logs will be passed to handlers and all of them will appear on stderr.
To log absolutely nothing on the screen you have to manually remove StreamHandler:
app.logger.handlers.pop(0)
and to log DEBUG and higher to the file set logging level also on the handler
log_handler.setLevel(logging.DEBUG)
Python logging is quite complicated. See this logging flow chart for better understanding what is going on :)
EDIT: to log only one specific level you need custom Filter object:
class LevelFilter:
def __init__(self, level):
self._level = level
def filter(self, log_record):
return log_record.levelno == self._level
log_handler.setLevel(logging.DEBUG)
log_handler.addFilter(LevelFilter(logging.DEBUG))
I found a few other posts on this but none that worked for me yet so I wanted to reach out and see if anyone could explain how to properly get / redirect / set handlers on some of the loggers present in Flask / Werkzeurg / sqlalchemy.
Research prior that could not answer my question:
https://github.com/pallets/flask/issues/1359
http://flask.pocoo.org/docs/dev/logging/
https://gist.github.com/ibeex/3257877
My configurations:
main.py
...
def init_app():
""" Runs prior to app launching, contains initialization code """
# set logging level
if not os.path.exists(settings.LOG_DIR):
os.makedirs(settings.LOG_DIR)
# default level
log_level = logging.CRITICAL
if settings.ENV == 'DEV':
log_level = logging.DEBUG
elif settings.ENV == 'TEST':
log_level = logging.WARNING
elif settings.ENV == 'PROD':
log_level = logging.ERROR
log_formatter = logging.Formatter("[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s")
api_logger = logging.getLogger()
api_handler = TimedRotatingFileHandler(
settings.API_LOG_FILE,
when='midnight',
backupCount=10
)
api_handler.setLevel(log_level)
api_handler.setFormatter(log_formatter)
api_logger.addHandler(api_handler)
logging.getLogger('werkzeug').addHandler(api_handler)
db_logger = logging.getLogger('sqlalchemy')
db_handler = TimedRotatingFileHandler(
settings.DB_LOG_FILE,
when='midnight',
backupCount=10
)
db_handler.setLevel(log_level)
db_handler.setFormatter(log_formatter)
db_logger.addHandler(db_handler)
logging.getLogger('sqlalchemy.engine').addHandler(db_handler)
logging.getLogger('sqlalchemy.dialects').addHandler(db_handler)
logging.getLogger('sqlalchemy.pool').addHandler(db_handler)
logging.getLogger('sqlalchemy.orm').addHandler(db_handler)
# add endpoints
...
if __name__ == '__main__':
init_app()
app.run(host='0.0.0.0', port=7777)
I tried grabbing and changes settings on the loggers a few different ways but I still end up with the werkzeug debug outputting to console and not my logs, I can see the logs are being created but it doesn't look like the loggers are actually outputting to them:
api.log (formatter wrote to it)
2018-02-15 12:03:03,944] {/usr/local/lib/python3.5/dist-packages/werkzeug/_internal.py:88} WARNING - * Debugger is active!
db.log (empty)
Any insight on this would be much appreciated!
UPDATE
I was able to get the werkzeug logger working using the long hand version, it seems the shorthand function calls shown were returning null objects. The sqlalchemy logger is still outputting to console though.. Could the engine configuration be overriding my filehandler?
main.py
...
# close current file handlers
for handler in copy(logging.getLogger().handlers):
logging.getLogger().removeHandler(handler)
handler.close()
for handler in copy(logging.getLogger('werkzeug').handlers):
logging.getLogger('werkzeug').removeHandler(handler)
handler.close()
for handler in copy(logging.getLogger('sqlalchemy.engine').handlers):
logging.getLogger('sqlalchemy.engine').removeHandler(handler)
handler.close()
for handler in copy(logging.getLogger('sqlalchemy.dialects').handlers):
logging.getLogger('sqlalchemy.dialects').removeHandler(handler)
handler.close()
for handler in copy(logging.getLogger('sqlalchemy.pool').handlers):
logging.getLogger('sqlalchemy.pool').removeHandler(handler)
handler.close()
for handler in copy(logging.getLogger('sqlalchemy.orm').handlers):
logging.getLogger('sqlalchemy.orm').removeHandler(handler)
handler.close()
# create our own custom handlers
log_formatter = logging.Formatter("[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s")
api_handler = TimedRotatingFileHandler(
settings.API_LOG_FILE,
when='midnight',
backupCount=10
)
api_handler.setLevel(log_level)
api_handler.setFormatter(log_formatter)
logging.getLogger().setLevel(log_level)
logging.getLogger().addHandler(api_handler)
logging.getLogger('werkzeug').setLevel(log_level)
logging.getLogger('werkzeug').addHandler(api_handler)
db_handler = TimedRotatingFileHandler(
settings.DB_LOG_FILE,
when='midnight',
backupCount=10
)
db_handler.setLevel(log_level)
db_handler.setFormatter(log_formatter)
logging.getLogger('sqlalchemy.engine').addHandler(db_handler)
logging.getLogger('sqlalchemy.engine').setLevel(log_level)
logging.getLogger('sqlalchemy.dialects').addHandler(db_handler)
logging.getLogger('sqlalchemy.dialects').setLevel(log_level)
logging.getLogger('sqlalchemy.pool').addHandler(db_handler)
logging.getLogger('sqlalchemy.pool').setLevel(log_level)
logging.getLogger('sqlalchemy.orm').addHandler(db_handler)
logging.getLogger('sqlalchemy.orm').setLevel(log_level)
database.py
...
engine = create_engine(getDBURI(), echo="debug", echo_pool=True, pool_recycle=10)
ANSWER
For future reference if anyone runs into this issue, sqlalchemy engine configuration echo=True|'debug' will OVERRIDE your loggers. Fixed the issue by changing my engine configuration to:
engine = create_engine(getDBURI(), echo_pool=True, pool_recycle=10)
And then everything worked like a charm. Cheers! :D
as i understand it your file based log configuration for werkzeug is actually working => it outputs into api.log
The db log handler is also working (file gets created etc.) but there is no output.
This is probably due to the loglevel of those loggers beeing on Error by default. You need to set them manually on a lower level like this:
logging.getLogger('sqlalchemy.engine').setLevel(logging.DEBUG)
logging.getLogger('sqlalchemy.dialects').setLevel(logging.DEBUG)
logging.getLogger('sqlalchemy.pool').setLevel(logging.DEBUG)
logging.getLogger('sqlalchemy.orm').setLevel(logging.DEBUG)
That werkzeug is still outputting to console is probably because there is allways a root logger defined. Before you add your new handlers you should do the following to remove all log handlers:
for handler in copy(logging.getLogger().handlers):
logging.getLogger().removeHandler(handler)
handler.close() # clean up used file handles
Then you can also assign your app log handler as the root log handler with
logging.getLogger().addHandler(api_handler)
If its not the root logger but just the werkzeug logger which has a default console logger defined you can also just remove all handlers from the werkzeug logger before adding yours like this:
for handler in copy(logging.getLogger('werkzeug').handlers):
logging.getLogger('werkzeug').removeHandler(handler)
handler.close() # clean up used file handles
logging.getLogger('werkzeug').addHandler(api_handler)