I seem to only be able to write to the Apache error log via stderr. Anyone know of a more structured logging architecture that I could use from my python web project, like commons?
This must have changed in the past four years. If you come across this question and want to do this then you can do it through the request object, i.e
def handler(req) :
req.log_error('Hello apache')
There isn't any built in support for mod_python logging to Apache currently. If you really want to work within the Apache logs you can check out this thread (make sure you get the second version of the posted code, rather than the first):
http://www.dojoforum.com/node/13239
http://www.modpython.org/pipermail/mod_python/2005-October/019295.html
If you're just looking to use a more structured logging system, the Python standard logging module referred to by Blair is very feature complete. Aside from the Python.org docs Blair linked, here's a more in-depth look at the module's features from onLamp:
http://www.onlamp.com/pub/a/python/2005/06/02/logging.html
And for a quickie example usage:
http://hackmap.blogspot.com/2007/06/note-to-self-using-python-logging.html
I've used the builtin Python logging module in (non-web) projects in the past, with success - it should work in a web-hosted environment as well.
I concur with Blair Conrad's post about the Python logging module. The standard log handlers sometimes drop messages however. It's worth using the logging module's SocketHandler and building a receiver to listen for messages and write them to file.
Here's mine: Example SocketHandler receiver.
Related
I have verbose logging and it looks like it is significantly impacting the performance of my Python program.
From what I've understood, it seems that by default, the Python logging module will buffer until one log message.
Is it possible to configure this buffer for several log messages? For example, can I make the Python logging module buffer up to 10 messages?
I have gone through the Python documentation and also stack overflow questions like this, this and this but there is no clear answer and therefore posting this.
Thank you for any input.
Use the MemoryHandler. It does exactly what you want. It buffers messages until some criteria are met and then sends them to another handler to be handled.
I use robot framework 3.0 under Python 2.7.8. Robot framework's documentation (http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#programmatic-logging-apis) states that
In addition to the new public logging API, Robot Framework offers a built-in support to Python's standard logging module. This works so that all messages that are received by the root logger of the module are automatically propagated to Robot Framework's log file.
I made a short library file to test this:
from logging import debug, error, info, warn
def try_logging():
info("This is merely a humble info message.")
debug("Most users never saw me.")
warn("I warn you about something.")
error("Something bad happened.")
My test case is:
*** Test Cases ***
Logtest
Try logging
When I run it it is a PASSED case, but nothing logged into the HTML log. The test execution log has the suit and the case and the keyword as it should but when I expand them nothing is logged but the "Start / End / Elapsed" line.
How could I forward the Python logger messages to Robot? As you can see the so called automatic propagation is not working automatically. My goal is to write a library that can be run with or without Robot Fw.
Ty for your help in advance.
After hours of code digging I managed to find the answer. I think it is worth sharing as it may be help you if you have some similar issue.
In my case I had some unused libraries imported. One of them was a class that was instantiated when Robot Framework imported the library file. This object had some logger settings that messed up the defaults, that is why I got no result in the robot log.
Without it I got the expected results and automatic propagation worked fine.
I'm writing a large hardware simulation library in Python3. For logging, I use the Python3 Logging module.
For controlling debug messages with method-level granularity, I learned "on the street" (ok, here at StackOverflow) to create sub-loggers within each method I wanted to log from:
sub_logger = logging.getChild("new_sublogger_name")
sub_logger.setLevel(logging.DEBUG)
# Sample debug message
sub_logger.debug("This is a debug message...")
By changing the call to setLevel(), the user is able to enable/disable debugging messages on a per-method basis.
Now the Boss Man don't like this approach. He's advocating a single-point at which all logging messages in the library can be enabled/disabled with the same method-level granularity. (This was to be accomplished by writing our own Python logging library BTW).
Not wanting to re-invent the logging wheel, I proposed to instead continue to use the Python Logging library, but instead use Filters to allow single-point control of logging messages.
Having not used Python Logging Filters very often, is there a consensus on using Filters vs Sublogger.setLevel() for this application? What are the pros/cons of each method?
I'm quite used to setLevel() after using it for a while, but that may be coloring my objectiveness. I DO NOT, however, wish to waste everyone's time writing another Python logging library.
I think the existing logging module does what you want. The trick is to separate the place where you call setLevel() (a configuration operation) from the places where you call getChild() (ongoing logging operations).
import logging
logger = logging.getLogger('mod1')
def fctn1():
logger.getChild('fctn1').debug('I am chatty')
# do stuff (notice, no setLevel)
def fctn2():
logger.getChild('fctn2').debug('I am even more chatty')
# do stuff (notice, no setLevel)
Notice there was no setLevel() there, which makes sense. Why call setLevel() every time and since when does a method know what logging level the user wants.
You set your logging levels in a configuration step at the beginning of the program. You can do it with the dictionary based configuration, a python module that does a bunch of setLevel() calls or even something you cook up with ini files or whatever. But basically it boils down to:
def config_logger():
logging.getLogger('abc.def').setLevel(logging.INFO)
logging.getLogger('mod1').setLevel(logging.WARN)
logging.getLogger('mod1.fctn1').setLeveL(logging.DEBUG)
(etc...)
Now, if you want to get fancy with filters, you can use them to inspect the stack frame and pull the method name out for you. But that gets more complicated.
I'm writing a web-application in Python, I haven't decided if I want to use Flask, web.py or something else yet, and I want to be able to do profile on the live application.
There seems to be very little information on how you go about implementing the instrumentation to do performance measurement, short of doing a lot of print datetime.now() everywhere.
What is the best way of going about instrumenting your Python application to allow good measurements to be made. I guess I'm looking for something similar to the Stackoverflow teams mvc-mini-profiler.
You could simply run cProfile tool that comes with Python:
python -m cProfile script.py
Of course, you would have to create the script.py file that would execute the parts of the code that you want to test. If you had some unit tests, you could also use that.
Or you couse use:
import cProfile
cProfile.run('foo()')
to profile it from foo entry point.
Amir Salihefendic wrote a short (150 LOC) RequestProfiler, which is described in this blog post:
http://amix.dk/blog/post/19359
I haven't tried it, but since it is a WSGI middleware, it should be somewhat pluggable.
You can just use a general purpose web application performance tool, such as httpperf. This works using an external client and works with any framework since it works against a standard interface (HTTP). Therefore it tests the full stack performance.
Use New Relic's Free monitoring system. You simply install an agent on the server and point to your flask init.py file. Once you run the application with proper agent setup, you will start seeing application metrics in see New Relic's online dashboard called APM.
By default it will show you graphs of your application's throughput (QPS/RPM), app response time, top transactions, error rate, error stack trace if any(eg for 500 error), calls to external services etc. In addition you can monitor your System stats too.
I have Django web-site working on tornado and nginx.
I took this tornado launcher script (tornading.py)
Then I'm using python openid that outputs some information to sys.stderr.
As a result I get IOError.
How can I redirect it using logging package?
I thought about
f = open("myfile.log", "w")
sys.stderr = f
or
python tornado.py > /dev/null 2>&1
But what is the best way to solve it?
The best way would be if the openid library didn't print to stderr, but used some kind of logging API instead (e.g. the logging module). I agree with thkala that modifying third-party code is not good in the long term, so you should fix it, and then provide the fix to the openid authors.
For the objective of advancing the open source community, that's the best way to solve it.
Using shell redirections is more of a work-around than a solution and it may not be always possible, depending on how the script is launched.
It has the distinct advantage, however, of you not having to modify third-party code. Local modifications - even minor ones - can become a major issue when you decide to e.g. update said code to its latest version from upstream.