I'm working with Django-nonrel on Google App Engine, which forces me to use logging.debug() instead of print().
The "logging" module is provided by Django, but I'm having a rough time using it instead of print().
For example, if I need to verify the content held in the variable x, I will put
logging.debug('x is: %s' % x). But if the program crashes soon after (without flushing the stream), then it never gets printed.
So for debugging, I need debug() to be flushed before the program exits on error, and this is not happening.
I think this may work for you, assuming you're only using one(or default) handler:
>>> import logging
>>> logger = logging.getLogger()
>>> logging.debug('wat wat')
>>> logger.handlers[0].flush()
It's kind of frowned upon in the documentation, though.
Application code should not directly instantiate and use instances of Handler. Instead, the Handler class is a base class that defines the interface that all handlers should have and establishes some default behavior that child classes can use (or override).
http://docs.python.org/2/howto/logging.html#handler-basic
And it could be a performance drain, but if you're really stuck, this may help with your debugging.
If the use case is that you have a python program that should flush its logs when exiting, use logging.shutdown().
From the python documentation:
logging.shutdown()
Informs the logging system to perform an orderly
shutdown by flushing and closing all handlers. This should be called
at application exit and no further use of the logging system should be
made after this call.
Django logging relies on the standard python logging module.
This module has a module-level method: logging.shutdown() which flushes all of the handlers and shuts down the logging system (i.e. logging can not longer be used after it is called)
Inspecting the code of this function shows that currently (python 2.7) the logging module holds a list of weak references to all handlers in a module-level variable called _handlerList so all of the handlers can be flushed by doing something like
[h_weak_ref().flush() for h_weak_ref in logging._handlerList]
because this solution uses the internals of the module #Mikes solution above is better, but it relies on having access to a logger, it can be generalized as follows:
[h.flush() for h in my_logger.handlerList]
a simple function that always working for register you debug messsages while programming. dont use it for production, since it will not rotate:
def make_log(message):
import datetime
with open('mylogfile.log','a') as f:
f.write(f"{datetime.datetime.now()} {message}\n")
then use as
make_log('my message to register')
when to put on production, just comment the last 2 lines
def make_log(message):
import datetime
#with open('mylogfile.log','a') as f:
# f.write(f"{datetime.datetime.now()} {message}\n")
Related
I use the scrapy library for scraping and it handels all the logging for me. The log file is very big and I was hoping for another way than to read through the whole file. I'm only interested in the exception.
Is it posible to get the raw logging.exception and read from that without creating another log file?
Use the python logging facility which is what Scrapy uses. At the program entry point you need to customize this logging facility, it can be in the __init__.py file for example or anything else before you use any log related calls.
What you need to do is set different handlers for different levels (ERROR, WARNING, ...). Exception level do not exist, you might be mistaking with Error or Critical levels.
This is the "Logging How To", you should check it.
Your solution would look like :
import logging
#this is the logger Scrapy uses throughout the app.
logger = logging.getLogger()
my_exception_handler = logging.FileHandler()
my_exception_handler.setLevel(Logging.ERROR)
logger.addHandler(my_exception_handler)
Finally, if you are really talking about exception and not error or critical levels. You'll need to inherit from your Exception class or overwrite it so you can use the same logging approach I just explained.
I run this earlier in the code
watch("httpstream")
Subsequently, any py2neo commands that triggers HTTP traffic will result in verbose logging. How can I stop the effect of watch() logging without creating a new Graph instance?
You don't. That is, I've not written a way to do that.
The watch function is intended only as a debugging utility for an interactive console session. You shouldn't need to use it in an application.
You can set the logging level to a higher value, for example
import logging
logging.getLogger("httpstream").setLevel(Logging.WARNING)
Get Logger Information
You can enumerate a list of all available loggers
print logging.Logger.manager.loggerDict.keys()
Then, you can use either
logging.getLogger("httpstream").getEffectiveLevel()
or
logging.getLogger("httpstream").isEnabledFor(logging.DEBUG)
to get the logging level.
My program is quite big and I want all its print statements to be logged so as a result I implemented
F = open('testy.txt','w')
sys.stdout = F
if app.button_press() == True and app.return_data():
data = app.return_data()
main(data[0],data[1],data[2],data[3],data[4],data[5],data[6],data[7],data[8])
F.close()
This is what I used to do with small programs with a few print statements but this program has few hundred of them I guess and as I run my program it freezes I think it has a lot of print statements and a memory overflow occurs therefore How could I log all my print statements to a .txt file without affecting the functionality of it?
You shouldn't be using print statements for logging, nor should you be redirecting standard out (or any other standard stream) in production code. Rather, you should be using the logging module to write messages, and setting up whatever handlers, probably a file handler or rotating file handler in your case, that you need to record your log messages to the right place.
If you already have too much existing code printing log messages to refactor in one sitting, I'd suggest implementing logging, using it for all logging going forward, setting up a file-like object that shunts standard out to logging to capture existing log messages, then refactoring out all your print-based logging over time.
I'm writing a large hardware simulation library in Python3. For logging, I use the Python3 Logging module.
For controlling debug messages with method-level granularity, I learned "on the street" (ok, here at StackOverflow) to create sub-loggers within each method I wanted to log from:
sub_logger = logging.getChild("new_sublogger_name")
sub_logger.setLevel(logging.DEBUG)
# Sample debug message
sub_logger.debug("This is a debug message...")
By changing the call to setLevel(), the user is able to enable/disable debugging messages on a per-method basis.
Now the Boss Man don't like this approach. He's advocating a single-point at which all logging messages in the library can be enabled/disabled with the same method-level granularity. (This was to be accomplished by writing our own Python logging library BTW).
Not wanting to re-invent the logging wheel, I proposed to instead continue to use the Python Logging library, but instead use Filters to allow single-point control of logging messages.
Having not used Python Logging Filters very often, is there a consensus on using Filters vs Sublogger.setLevel() for this application? What are the pros/cons of each method?
I'm quite used to setLevel() after using it for a while, but that may be coloring my objectiveness. I DO NOT, however, wish to waste everyone's time writing another Python logging library.
I think the existing logging module does what you want. The trick is to separate the place where you call setLevel() (a configuration operation) from the places where you call getChild() (ongoing logging operations).
import logging
logger = logging.getLogger('mod1')
def fctn1():
logger.getChild('fctn1').debug('I am chatty')
# do stuff (notice, no setLevel)
def fctn2():
logger.getChild('fctn2').debug('I am even more chatty')
# do stuff (notice, no setLevel)
Notice there was no setLevel() there, which makes sense. Why call setLevel() every time and since when does a method know what logging level the user wants.
You set your logging levels in a configuration step at the beginning of the program. You can do it with the dictionary based configuration, a python module that does a bunch of setLevel() calls or even something you cook up with ini files or whatever. But basically it boils down to:
def config_logger():
logging.getLogger('abc.def').setLevel(logging.INFO)
logging.getLogger('mod1').setLevel(logging.WARN)
logging.getLogger('mod1.fctn1').setLeveL(logging.DEBUG)
(etc...)
Now, if you want to get fancy with filters, you can use them to inspect the stack frame and pull the method name out for you. But that gets more complicated.
I have a Django app on a Linux server. In one of the views, some form of print command is executed, and some string gets printed. How can I find out what the printed string was? Is there some log in which these things are kept?
The output should be in the terminal, where django was started. (if you don't started it directly, I don't believe there's a way to read it)
As linkedlinked pointed out, it's the best to not use print, because this can cause Exceptions! But that's not the only reason: There are modules (like logging) made for such purposes and they have a lot more options.
This site (even when it's from 2008) confirm my statements:
If you want to know what’s going on inside a view, the quickest way is to drop in a print statement. The development server outputs any print statements directly to the terminal; it’s the server-side alternative to a JavaScript alert().
If you want to be a bit more sophisticated with your logging, it’s worth turning to Python’s logging module (part of the standard library). You can configure it in your settings.py: here he describes, what to do (look on the site)
For debugging-purposes you could also enable the debug-mode or use the django-debug-toolbar.
Hope it helps! :)
Never use print, as once you deploy, it will print to stdout and WGSI will break.
Use the logging. For development purposes, is really easy to setup. On your project __init__.py:
import logging
from django.conf import settings
fmt = getattr(settings, 'LOG_FORMAT', None)
lvl = getattr(settings, 'LOG_LEVEL', logging.DEBUG)
logging.basicConfig(format=fmt, level=lvl)
logging.debug("Logging started on %s for %s" % (logging.root.name, logging.getLevelName(lvl)))
Now everything you log goes to stderr, in this case, your terminal.
logging.debug("Oh hai!")
Plus you can control the verbosity on your settings.py with a LOG_LEVEL setting.
The print shows up fine with "./manage.py runserver" or other variations - like Joschua mentions, it shows up in the terminal where you started it. If you're running FCGI from cron or such, that just gets dumped into nothingness and you lose it entirely.
For places where I want "print" like warnings or notices to come out, I use an instance of python's logger that pushes to syslog to capture the output and put it someplace. I instantiate an instance of logging in one of the modules as it gets loaded - models.py was the place I picked, just for its convenience and I knew it would always get evaluated before requests came rolling in.
import logging, logging.handlers
logger = logging.getLogger("djangosyslog")
hdlr = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_DAEMON)
formatter = logging.Formatter('%(filename)s: %(levelname)s: %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
Then when you want to invoke a message to the logger in your views or whatever:
logger = logging.getLogger("djangosyslog")
logging.warning("Protocol problem: %s", "connection reset", extra=d)
There's .error(), .critical(), and more - check out http://docs.python.org/library/logging.html for the nitty gritty details.
Rob Hudson's debug toolbar is great if you're looking for that debug information - I use it frequently in development myself. It gives you data about the current request and response, including the SQL used to generate any given page. You can inject into that data like a print by shoving the
strings you're interested into the context/response - but I found that to be a bit difficult to deal with.
A warning: if you try to deploy code with print statements under WSGI, expect things to break. Use the logging module instead.
If you are using apache2 server to run django application and enabled access & error logs, your print statements will be printed in the error logs.
While you running your application kindly do the following as root user in linux,
tail -f /path-to-error-file.log
mostly apache2 logs will be in this location /var/log/apache2/.
It will print when ever it finds print command in your function.