Develop Reference

Python - How to log records to both console and stdout

I'm trying to replace all print statements from the old test framework with Python logging.
I need to use a simple logging.debug('something') way to write records to both console and test report.
I'm using a custom logger for each test module like this:
logger.py
import logging
from sys import stdout
def setup_logger(filename):
logger = logging.getLogger(filename)
logger.setLevel(logging.DEBUG)
stdout_handler = logging.StreamHandler(stdout)
logger.addHandler(stdout_handler)
return logger
test_file.py
from logger import setup_logger
logger = setup_logger(__name__)
logger.debug('something')
Currently I can write to the console but I don't see those records in sys.stdout when I'm generating test reports. I'm using HtmlTestRunner for generating test reports and it looks like it doesn't receive stdout records from tests here:
self._stdout_data = sys.stdout.getvalue()
https://github.com/oldani/HtmlTestRunner/blob/master/HtmlTestRunner/result.py#L181
I'm not very familiar with logging and I don't understand what might be an issue here.
Is sys.stdout something completely different from the stdout where my custom logger writes messages?
Is there an easy way to solve this issue?
I read other similar questions about this (so my custom logger was written after reading those) but my issue is a bit different as I'm using the report generator which is parsing sys.stdout.
UPD: Or maybe I'm already doing this right but I need to read not the sys.stdout but something else to find my records when generating test report?
UPD2: Found a similar old question which was never answered) Maybe there's no quick solution for that(
View passed testcases and show log messages in html report when using python html-testrunner??

It looks like the only good way to solve such an issue is writing a function which does 2 things: adds logs to console (I ended up with just using loguru - simple and clear) and prints the same message (the only purpose of that is to support old test reports). Of course it might look weird that someone needs output data in success reports as well but it's not my personal choice)

Python, read log exception from logging.exception

I use the scrapy library for scraping and it handels all the logging for me. The log file is very big and I was hoping for another way than to read through the whole file. I'm only interested in the exception.
Is it posible to get the raw logging.exception and read from that without creating another log file?

Use the python logging facility which is what Scrapy uses. At the program entry point you need to customize this logging facility, it can be in the __init__.py file for example or anything else before you use any log related calls.
What you need to do is set different handlers for different levels (ERROR, WARNING, ...). Exception level do not exist, you might be mistaking with Error or Critical levels.
This is the "Logging How To", you should check it.
Your solution would look like :
import logging
#this is the logger Scrapy uses throughout the app.
logger = logging.getLogger()
my_exception_handler = logging.FileHandler()
my_exception_handler.setLevel(Logging.ERROR)
logger.addHandler(my_exception_handler)
Finally, if you are really talking about exception and not error or critical levels. You'll need to inherit from your Exception class or overwrite it so you can use the same logging approach I just explained.

How do I define a different logger for an imported module in Python?

I'm using Advanced Python Scheduler in a Python script. The main program defines a log by calling logging.basicConfig with the file name of the log that I want. This log is also set to "DEBUG" as the logging level, since that's what I need at present for my script.
Unfortunately, because logging.basicConfig has been set up in this manner, apscheduler writes its log entries to the same log file. There are an awful lot of these, especially since I have one scheduled task that runs every minute.
Is there any way to redirect apscheduler's log output to another log file (without changing apscheduler's code) while using my log file for my own script? I.e. is there a way to change the file name for each module's output within my script?
I tried reading the module page and the HOWTO for logging, but could not find an answer to this.

Set the logger level for apscheduler to your desired value (e.g. WARNING to avoid seeing DEBUG and INFO messages from apscheduler like this:
logging.getLogger('apscheduler').setLevel(logging.WARNING)
You will still get messages for WARNING and higher severities. To direct messages from apscheduler into a separate file, use
aplogger = logging.getLogger('apscheduler')
aplogger.propagate = False
aplogger.setLevel(logging.WARNING) # or whatever
aphandler = logging.FileHandler(...) # as per what you want
aplogger.addHandler(aphandler)
Ensure the above code is only called once (otherwise you will add multiple FileHandler instances - probably not what you want).

maybe you want to call logging.getLogger("apscheduler") and setup its log file in there? see this answer https://stackoverflow.com/a/2031557/782168

How can I mirror Django logging to a second file temporarily?

I want to create task-specific log files that reflect what's happening during a particular operation, but I want these logging messages to also go to the primary Django log.
My current solution, which seems to work fine at first glance, is something like:
logger = getLogger("%s:%s" % (__name__, task_id))
handler = FileHandler(task_log_file)
logger.addHandler(handler)
# Work
logger.removeHandler(handler)
As I said, this works, but the main issue that occurs to me is that this logger isn't really temporary -- from what I've read of logging.Manager each logger will just hang around indefinitely until shutdown. In this case, when I'm done I know I won't use the logger again (okay, technically I might, but that will be rare), and assuming the system is stable this could be running through hundreds of thousands of tasks.
Is there a "right" way to do this?

You could have a way to mark-and-sweep your logging resource files, or just use a singleton pattern.

python logging question

I have a question about python standard logging mechanism. So if I use logging.config.fileConfig to load my configuration file, then I create loggers for some modules using logging.getLogger test them right after creation and they work. Now if I call logging.config.fileConfig again with the same configuration file and create loggers for some other module would the previos ones still work ? Basically for the following logic:
logging.config.fileConfig(config_file)
logger1 = logging.getLogger(module1)
logger2 = logging.getLogger(module2)
logging.config.fileConfig(config_file)
logger3 = logging.getLogger(module3)
config_file is the same in both calls. Should logger1 and logger2 be functional ? How about if config_file is different in those calls? Currently my logger1 and logger2 are not working after i load a new config_file. So first step is to check if this is normal behaviour. If so is it possible to make this work without merging the two config_files into one big one?
Regards,
Bogdan

Config files are intended to completely replace the existing configuration with whatever is in the confguration - any loggers which are not named in the configuration, or children thereof, are disabled by fileConfig(), as documented here. you can prevent this disabling, but only on recent Python versions. It's not generally good practice to call fileConfig() multiple times in a program, unless you have a specific need to do so. It's not forbidden, but it's not usual.
A common usage involves configuring handlers on the root logger and perhaps one or two top-level loggers; does this apply to you?