In a python project with multiple threads my logging works well to write to a logger file. Basically based on Logging, StreamHandler and standard streams
Part of my project is a bottle web server which runs well also. But every bottle call writes a log to the console like this:
192.168.178.20 - - [26/Jun/2015 20:22:17] "GET /edit?addJob HTTP/1.1" 200 48028
How to handle this the same way as with the other code, so the bottle logs go also to the logger file?
If you're rolling your own solution, you should write a simple Bottle plugin that emits log lines to a logging logger. Here's an example that sets up a basic logger, defines the logging plugin, and creates a Bottle app with that plugin installed on all routes.
from bottle import Bottle, request, response
from datetime import datetime
from functools import wraps
import logging
logger = logging.getLogger('myapp')
# set up the logger
logger.setLevel(logging.INFO)
file_handler = logging.FileHandler('myapp.log')
formatter = logging.Formatter('%(msg)s')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
def log_to_logger(fn):
'''
Wrap a Bottle request so that a log line is emitted after it's handled.
(This decorator can be extended to take the desired logger as a param.)
'''
#wraps(fn)
def _log_to_logger(*args, **kwargs):
request_time = datetime.now()
actual_response = fn(*args, **kwargs)
# modify this to log exactly what you need:
logger.info('%s %s %s %s %s' % (request.remote_addr,
request_time,
request.method,
request.url,
response.status))
return actual_response
return _log_to_logger
app = Bottle()
app.install(log_to_logger)
#app.route('/')
def home():
return ['hello, world']
app.run(host='0.0.0.0', port='8080', quiet=True)
Running that code yields what you want:
% python myapp.py &
% curl -v http://localhost:8080/
% tail myapp.log
127.0.0.1 2015-06-27 16:57:09.983249 GET http://localhost:8080/ 200 OK
I'm trying to use Ron's solution with starting the bottle program on a thread:
tWeb = Thread(target=runWeb, args=('192.168.178.16', 5003)).start()
with
def runWeb(aserver, aport):
run(host=aserver, port=aport, debug=True)
but that fails. Any 'print' goes to the file, but not the 'yield' (see above), it goes to the console.
Also changing "debug=True" to "quiet=True" only changes to: there is no output on the console at all.
You running builtin server right ? Then you can make a simple plugin :
from bottle import request, response, route, install, run
from datetime import datetime
def logger(func):
def wrapper(*args, **kwargs):
log = open('log.txt', 'a')
log.write('%s %s %s %s %s \n' % (request.remote_addr, datetime.now().strftime('%H:%M'),
request.method, request.url, response.status))
log.close()
req = func(*args, **kwargs)
return req
return wrapper
install(logger)
#route('/')
def index():
return 'Hello, World'
run(quiet=True)
Or try this one
Related
I have a logger defined as in logger.py:
def my_logger(module_name):
logger = logging.getLogger(module_name)
logger.setLevel(logging.DEBUG)
# stream handler
# debug handler
f1_handler = logging.handlers.RotatingFileHandler(os.path.join(abspath, "logs/debug.log"))
f1_handler.setLevel(logging.DEBUG)
f1_handler.setFormatter(format)
# warning handler
f3_handler = logging.handlers.RotatingFileHandler(os.path.join(abspath, "logs/warning.log"))
f3_handler.setLevel(logging.WARNING)
f3_handler.setFormatter(format)
# error handler
f4_handler = logging.handlers.RotatingFileHandler(os.path.join(abspath, "logs/error.log"))
f4_handler.setLevel(logging.ERROR)
f4_handler.setFormatter(format)
# Add handlers to the logger
logger.addHandler(f1_handler)
logger.addHandler(f3_handler)
logger.addHandler(f4_handler)
return logger
Then in my test.py, I can test the logger as test.py:
from kg import logger
logger = logger.my_logger(__name__)
if __name__ == "__main__":
logger.info('Start main ...')
This works as expected, the logger message only goes to debug.log file under my logs/debug.log, meaning that it doesn't print the message to console.
However, if I test it in a Flask application, and in browser I sent a request, it always print the log message to console as well, in addition to the log file.
My flask api in flask_test.py:
from search import keyword_match
#api.route('/match', methods=['GET'])
def match():
text, parameters = _parse_parameters()
json_result = keyword_match(text, **parameters)
return json_result
if __name__ == '__main__':
#api.run(host='0.0.0.0', port='5000')
logger.info('Test log in Flask api!')
from waitress import serve
serve(api, host="0.0.0.0", port=5000)
The logger here in the 'main' still works as expected, it doesn't print out to console, since no stream handler is configured in my logger. However, there is a logger message in search.py, which defines the keyword_match function:
from kg import logger
logger = logger.my_logger(__name__)
def keyword_match(keyword, **args):
logger.info('keyword: {}'.format(keyword))
...
It is in this file, it always prints out the log info onto console even if there is no stream handler in my own logger definition. It seems Flask stream handler is always turned on and how to turn it off? I tried this in my flask_test.py, but it didn't helper:
logger = logger.my_logger(__name__)
app = Flask(__name__)
from flask.logging import default_handler
app.logger.removeHandler(default_handler)
What's the problem?
Have you tried moving/removing this line of code: logger = logger.my_logger(__name__) to after you remove the default handler?
From my understanding, flask has a default understanding of what the logger object is. But having that line of code before you reference app.logger means that the meaning of logger has been changed.
app = Flask(__name__)
from flask.logging import default_handler
app.logger.removeHandler(default_handler)
logger = logger.my_logger(__name__)
I want to make log files when I run my minimal flask app, basically, I want to log all logging messages from the python console when different endpoints are used. To do so, I used a flask logging mechanism but couldn't catch logging messages including a running sessions on python concole, logging info from testing different endpoints with the specific routes. How can I catch running session info from python console? How do I make log for errors, warning, session info from python console? Is there any way to make this happen? any thought?
my current attempt
here is my current attempt which failed to catch log info on python console:
import os, logging, logging.handlers, traceback
from flask import Flask, jsonify, request
from flask_restplus import Api, Resource, Namespce,fields, reqparse, inputs
def primes_method1(n):
out = list()
for num in range(1, n+1):
prime = True
for i in range(2, num):
if (num % i == 0):
prime = False
if prime:
out.append(num)
return out
def getLogger(logname, logdir, logsize=500*1024, logbackup_count=4):
if not os.path.exists(logdir):
os.makedirs(logdir)
logfile='%s/%s.log' % (logdir, logname)
loglevel = logging.INFO
logger = logging.getLogger(logname)
logger.setLevel(loglevel)
if logger.handlers is not None and len(logger.handlers) >= 0:
for handler in logger.handlers:
logger.removeHandler(handler)
logger.handlers = []
loghandler = logging.handlers.RotatingFileHandler(
logfile, maxBytes=logsize, backupCount=logbackup_count)
formatter = logging.Formatter('%(asctime)s-%(name)s-%(levelname)s-%(message)s')
loghandler.setFormatter(formatter)
logger.addHandler(loghandler)
return logger
app = Flask(__name__)
api = Api(app)
ns = api.namespace('ns')
payload = api.model('Payload', {
'num': fields.Integer(required=True)
})
logger = getLogger('testLog', '~/')
#ns.route('/primenumber')
class PrimeResource(Resource):
#ns.expect(payload)
def post(self):
logger.info("get prime number")
param = request.json['num']
try:
res = primes_method1(param)
return jsonify({'output': res})
except:
return None, 400
ns1 = Namespce('')
#ns1.route('/log')
class logResource(Resource):
def get(self):
logger.info("return saved all logers from above endpoints")
if __name__ == '__main__':
api.add_namespace(ns)
api.add_namespace(ns1)
app.run(debug=True)
basically, when I test endpoint with sample data, I want to see all logged messages at #ns1.route('/log') endpoint. In my attempt, I couldn't catch running session info on python console. How to log flask running session info on python console? Is there any way to do this?
Your question isn't really related to Flask, unless I'm overlooking something unique to Flask that doesn't conform to the logging interface. This isn't a workaround either, just normal usage of the tools.
Try to add a handler to your logger like so:
loghandler = logging.handlers.RotatingFileHandler(
logfile, maxBytes=logsize, backupCount=logbackup_count)
formatter = logging.Formatter('%(asctime)s-%(name)s-%(levelname)s-%(message)s')
loghandler.setFormatter(formatter)
logger.addHandler(loghandler)
loghandler2 = logging.StreamHandler()
loghandler2.setFormatter(formatter)
logger.addHandler(loghandler2)
return logger
I have the following script that I want only the "DEBUG" log messages to be logged to the file, and nothing to the screen.
from flask import Flask, request, jsonify
from gevent.pywsgi import WSGIServer
import usaddress
app = Flask(__name__)
from logging.handlers import RotatingFileHandler
import logging
#logging.basicConfig(filename='error.log',level=logging.DEBUG)
# create a file to store weblogs
log = open('error.log', 'w'); log.seek(0); log.truncate();
log.write("Web Application Log\n"); log.close();
log_handler = RotatingFileHandler('error.log', maxBytes =1000000, backupCount=1)
formatter = logging.Formatter(
"[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s"
)
log_handler.setFormatter(formatter)
app.logger.setLevel(logging.DEBUG)
app.logger.addHandler(log_handler)
#app.route('/')
def hello():
return "Hello World!"
#app.route("/parseAddress", methods=["POST"])
def parseAddress():
address = request.form['address']
return jsonify(usaddress.tag(address)), 200
if __name__ == '__main__':
# app.run(host='0.0.0.0')
http_server = WSGIServer(('', 5000), app, log=app.logger)
http_server.serve_forever()
But right now even "INFO" messages are being logged to the file and to the screen. How can I have only the "DEBUG" messages logged to the file and nothing to the screen?
I did some interactive testing and it looks like app.logger is a logger named with python file
print(app.logger.name) # filename
and it has one handler
print(app.logger.handlers) # [<StreamHandler <stderr> (NOTSET)>]
A handler with level logging.NOTSET processes all messages (from all logging levels). So when you set app.logger.setLevel(logging.DEBUG) then all debug and higher logs will be passed to handlers and all of them will appear on stderr.
To log absolutely nothing on the screen you have to manually remove StreamHandler:
app.logger.handlers.pop(0)
and to log DEBUG and higher to the file set logging level also on the handler
log_handler.setLevel(logging.DEBUG)
Python logging is quite complicated. See this logging flow chart for better understanding what is going on :)
EDIT: to log only one specific level you need custom Filter object:
class LevelFilter:
def __init__(self, level):
self._level = level
def filter(self, log_record):
return log_record.levelno == self._level
log_handler.setLevel(logging.DEBUG)
log_handler.addFilter(LevelFilter(logging.DEBUG))
Bottle.py ships with an import to handle throwing HTTPErrors and route to a function.
Firstly, the documentation claims I can (and so do several examples):
from bottle import error
#error(500)
def custom500(error):
return 'my custom message'
however, when importing this statement error is unresolved but on running the application ignores this and just directs me to the generic error page.
I found a way to get around this by:
from bottle import Bottle
main = Bottle()
#Bottle.error(main, 500)
def custom500(error):
return 'my custom message'
But this code prevents me from embedding my errors all in a separate module to control the nastiness that would ensue if I kept them in my main.py module because the first argument has to be a bottle instance.
So my questions:
Has anyone else experienced this?
why doesn't error seem to resolve in only my case (I installed from pip install bottle)?
Is there a seamless way to import my error routing from a separate python module into the main application?
If you want to embed your errors in another module, you could do something like this:
error.py
def custom500(error):
return 'my custom message'
handler = {
500: custom500,
}
app.py
from bottle import *
import error
app = Bottle()
app.error_handler = error.handler
#app.route('/')
def divzero():
return 1/0
run(app)
This works for me:
from bottle import error, run, route, abort
#error(500)
def custom500(error):
return 'my custom message'
#route("/")
def index():
abort("Boo!")
run()
In some cases I find it's better to subclass Bottle. Here's an example of doing that and adding a custom error handler.
#!/usr/bin/env python3
from bottle import Bottle, response, Route
class MyBottle(Bottle):
def __init__(self, *args, **kwargs):
Bottle.__init__(self, *args, **kwargs)
self.error_handler[404] = self.four04
self.add_route(Route(self, "/helloworld", "GET", self.helloworld))
def helloworld(self):
response.content_type = "text/plain"
yield "Hello, world."
def four04(self, httperror):
response.content_type = "text/plain"
yield "You're 404."
if __name__ == '__main__':
mybottle = MyBottle()
mybottle.run(host='localhost', port=8080, quiet=True, debug=True)
I'm searching a wsgi middleware which I can warp around a wsgi applications and which lets me monitor incoming and outgoing http requests and header fields.
Something like firefox live headers, but for the server side.
The middleware
from wsgiref.util import request_uri
import sys
def logging_middleware(application, stream=sys.stdout):
def _logger(environ, start_response):
stream.write('REQUEST\n')
stream.write('%s %s\n' %(
environ['REQUEST_METHOD'],
request_uri(environ),
))
for name, value in environ.items():
if name.startswith('HTTP_'):
stream.write(' %s: %s\n' %(
name[5:].title().replace('_', '-'),
value,
))
stream.flush()
def _start_response(code, headers):
stream.write('RESPONSE\n')
stream.write('%s\n' % code)
for data in headers:
stream.write(' %s: %s\n' % data)
stream.flush()
start_response(code, headers)
return application(environ, _start_response)
return _logger
The test
def application(environ, start_response):
start_response('200 OK', [
('Content-Type', 'text/html')
])
return ['Hello World']
if __name__ == '__main__':
logger = logging_middleware(application)
from wsgiref.simple_server import make_server
httpd = make_server('', 1234, logger)
httpd.serve_forever()
See also the werkzeug debugger Armin wrote, it's usefull for interactive debugging.
That shouldn't be too hard to write yourself as long as you only need the headers. Try that:
import sys
def log_headers(app, stream=None):
if stream is None:
stream = sys.stdout
def proxy(environ, start_response):
for key, value in environ.iteritems():
if key.startswith('HTTP_'):
stream.write('%s: %s\n' % (key[5:].title().replace('_', '-'), value))
return app(environ, start_response)
return proxy
If you want Apache-style logs, try paste.translogger
But for something more complete, though not in a very handy or stable location (maybe copy it into your source) is wsgifilter.proxyapp.DebugHeaders
And writing one using WebOb:
import webob, sys
class LogHeaders(object):
def __init__(self, app, stream=sys.stderr):
self.app = app
self.stream = stream
def __call__(self, environ, start_response):
req = webob.Request(environ)
resp = req.get_response(self.app)
print >> self.stream, 'Request:\n%s\n\nResponse:\n%s\n\n\n' % (req, resp)
return resp(environ, start_response)
The mod_wsgi documentation provides various tips on debugging which are applicable to any WSGI hosting mechanism and not just mod_wsgi. See:
http://code.google.com/p/modwsgi/wiki/DebuggingTechniques
This includes an example WSGI middleware that captures request and response.
My WebCore project has a bit of middleware that logs the entire WSGI environment (thus Beaker sessions, headers, etc.) for the incoming request, headers for outbound responses, as well as performance information to a MongoDB database. Average overhead is around 4ms.
The module has been removed from the core package, but hasn’t yet been integrated into its own. The current version as of this answer is available in the Git history:
http://github.com/GothAlice/WebCore/blob/cd1d6dcbd081323869968c51a78eceb1a32007d8/web/extras/cprofile.py