I am setting up my Python logging in main.py via reading in a file and using the fileConfig option. I want to be able to switch between testing and live logging configurations, so I want to read in a separate config file first and extract the logging config file path from there.
The problem here is that other files that I import from main.py grab their own logger via log = getLogger(__name__), and this happens at import time. These links then get broken when the new configuration is loaded in, and these modules end up without the logging working the way I expect.
I can't easily delay the importing of these modules without a lot of refactoring, so is there any other way of being able to keep this method of setting up loggers by module name while still loading in the log configuration later?
I'm not sure from your question exactly how things are breaking, but here's how I see it. The various modules which do log = logging.getLogger(__name__) will have valid names for their loggers (logger name = package name), unless you were to somehow actually move the modules to some other package location.
At import time, the logging configuration may or may not have been set, and there shouldn't be any actual logging calls made as a side-effect of the import (if there are, the messages may have nowhere to go).
Loading a new configuration using fileConfig typically just sets handlers, formatters and levels on loggers.
When you subsequently call code in the imported modules, they log via their loggers, which have handlers attached by your previous configuration call - so they will output according to the configuration.
You should be aware that on older versions of Python (<= 2.5), calls to fileConfig would unavoidably disable existing loggers which weren't named in the configuration - in more recent versions of Python (>= 2.6), this is configurable using a disable_existing_loggers=False keyword argument passed to fileConfig. You may want to check this, as it sometimes leads to unexpected behaviour (the default for that parameter is True, for compatibility with behaviour under the older Python versions).
If you post more details about what seems broken, I might be able to provide a better diagnosis of what's going on.
Related
Hi this is hopefully a basic question. I have a python library with a lot of logger messages such as this:
log = logging.getLogger(__name__)
log.info("blah")
log.warning("blah")
...
Then I have a separate code that imports and runs this library. In that code, I added this, thinking it would cause all logging messages to go to this file:
log = logging.getLogger(__name__)
fh = logging.FileHandler("/some/file/location/log.txt")
log.addHandler(fh)
This does successfully pass all log messages in that script to direct to that file, but the logging messages from the library aren't being passed along. I don't want to specify the file path from within the library, that doesn't make much sense, I want it specified in the code that runs the library. Most of the examples I'm seeing show imports happening with parent/child modules, but what about one module that calls a completely different module? Does my library need to accept a logger as an argument to use, or can I use the logging module to handle this?
Thanks.
Looks like you are creating two instances of the Logger class. Only the instance in your script is being configured to write to the file location.
Each time the 'getLogger' method is called, it provides a reference to the Logger with the specified name. If a Logger with that name doesn't exist, a new one is created. Note that in Python, __name__ specifies the module name. Since you are calling the library from a script, I'd assume you have two separate modules, hence two different Loggers.
For a quick-and-dirty approach, you can use:
import my_library
log = logging.getLogger(my_library.__name__)
fh = logging.FileHandler("/some/file/location/log.txt")
log.addHandler(fh)
Where my_library is your newly defined library. This will provide the logger which the library instantiated, instead of creating a new one.
Another approach would be to define a module-level function like this:
# In your script
import my_library
log_location = "/some/file/location/log.txt"
my_library.set_log_location(log_location)
# In your newly defined library
def set_log_location(path):
log = logging.getLogger(__name__)
fh = logging.FileHandler("/some/file/location/log.txt")
log.addHandler(fh)
The second approach wouldn't require the user knowing that your library uses the logging module.
In your application, e.g. in the if __name__ == '__main__' clause, configure the root logger with the handlers you want, using e.g.
logging.basicConfig(level=logging.DEBUG,
filename='/some/file/location/log.txt',
filemode='w',
format='%(asctime)s %(message)s')
and then all logging from your application, your libraries and third-party libraries should write log messages to the file. The sources of logged events (application or libraries) don't need to know or care where the events they log end up - that's taken care of by the configuration. If you need more involved configuration than basicConfig() provides, you can use logging's dictConfig() API to configure logging.
I have multiple modules, which are called by a principal script. Each one does log messages using logging Python builtin package.
How can I log an session ID, set during the execution of the main script, across all modules, without needing to push this variable in each module?
I have set up a python configuration file, called config, with:
import logging
logging.basicConfig(
level=logging.DEBUG,
format="%(filename)s:%(lineno)s|%(funcName)3s()|%(asctime)s|%(levelname)s|%(message)s",
handlers=[
logging.FileHandler("debug.log"),
logging.StreamHandler()
]
)
Other modules are using this pre configured logging object, via import, so I am using this:
from config import logging
But I need to log an ID from my current session, my log should look like:
module_name.py:25|function_name()|2020-04-27 18:28:26,518|INFO|Session_ID=abc123|some_message_here
I have tried to put this variable in the config file, set it, and then use it in a function named "log_info" and "log_debug" in this file, but my output log does not trace python script name and function name any more.
Does anyone knows how to handle this situation?
I don't know if this is the 100% best solution, or for sure if it'll work in your situation, but I used it for something similar (verbosity printing levels that would persist across several scripts).
Create a Python file, named, say, sessionid. Inside that, define a top-level setting named id. To set your id, import sessionid and sessionid.id = 'some_id'. Then have your config file import sessionid as well and use sessionid.id as needed.
It took me a bit to figure out that I had to access it like that; changing it using from sessionid import id; id = 'some_id' only persists within the script that does so.
It would make sense to put the id variable in your config file, but only if you don't need to set it in the same script that also needs to from config import logging.
I'm writing a large hardware simulation library in Python3. For logging, I use the Python3 Logging module.
For controlling debug messages with method-level granularity, I learned "on the street" (ok, here at StackOverflow) to create sub-loggers within each method I wanted to log from:
sub_logger = logging.getChild("new_sublogger_name")
sub_logger.setLevel(logging.DEBUG)
# Sample debug message
sub_logger.debug("This is a debug message...")
By changing the call to setLevel(), the user is able to enable/disable debugging messages on a per-method basis.
Now the Boss Man don't like this approach. He's advocating a single-point at which all logging messages in the library can be enabled/disabled with the same method-level granularity. (This was to be accomplished by writing our own Python logging library BTW).
Not wanting to re-invent the logging wheel, I proposed to instead continue to use the Python Logging library, but instead use Filters to allow single-point control of logging messages.
Having not used Python Logging Filters very often, is there a consensus on using Filters vs Sublogger.setLevel() for this application? What are the pros/cons of each method?
I'm quite used to setLevel() after using it for a while, but that may be coloring my objectiveness. I DO NOT, however, wish to waste everyone's time writing another Python logging library.
I think the existing logging module does what you want. The trick is to separate the place where you call setLevel() (a configuration operation) from the places where you call getChild() (ongoing logging operations).
import logging
logger = logging.getLogger('mod1')
def fctn1():
logger.getChild('fctn1').debug('I am chatty')
# do stuff (notice, no setLevel)
def fctn2():
logger.getChild('fctn2').debug('I am even more chatty')
# do stuff (notice, no setLevel)
Notice there was no setLevel() there, which makes sense. Why call setLevel() every time and since when does a method know what logging level the user wants.
You set your logging levels in a configuration step at the beginning of the program. You can do it with the dictionary based configuration, a python module that does a bunch of setLevel() calls or even something you cook up with ini files or whatever. But basically it boils down to:
def config_logger():
logging.getLogger('abc.def').setLevel(logging.INFO)
logging.getLogger('mod1').setLevel(logging.WARN)
logging.getLogger('mod1.fctn1').setLeveL(logging.DEBUG)
(etc...)
Now, if you want to get fancy with filters, you can use them to inspect the stack frame and pull the method name out for you. But that gets more complicated.
My project consists of a number of namespaced packages and I want to set up logging properly for them: they are meant to be used as a library by other "frontends".
Suppose I have the case, for package foo.xyz:
foo/
__init__.py
xyz/
__init__.py
bar.py
baz.py
My idea would be to retain information from where the log is being generated, so for example in bar.py
import logging
log = logging.getLogger(__name__)
log.addHandler(logging.NullHandler()) # Python 2.7
log.setLevel(....)
However I'm not sure how to call this from the frontend (which imports several bits from different packages) to display everything without hassle. For example, I'm using foo.abc and foo.xyz, set up like above for logging. I would like to use propagation, but currently this doesn't work:
from foo.xyz import bar
from foo.abc import baz
log = logging.getLogger()
log.addHandler(logging.StreamHandler())
log.setLevel(logging.DEBUG)
do_my_stuff()
However, no output is being generated from the library's loggers. What am I doing wrong?
EDIT: So far I can get output if I get the logger corresponding to the parent module's namespace:
log = logging.getLogger("foo.xyz")
However I'm trying to grab everything in one call: I wonder if I can do that since, as I wrote earlier, this set of packages uses a namespace.
You don't need to add a NullHandler to all sub-packages of foo - you can just add the NullHandler to the foo logger, which could be done in foo/__init__.py.
The NullHandler is only added to library code to handle situations where the library is used but logging isn't configured by the using application. Thomas Vander Stichele is wrong to state that adding a NullHandler would cause messages to be dropped.
In your application, you can configure logging as you wish - whatever handlers, levels etc. Level setting should not (in general) be done in the modules themselves, but in some central place, typically called similarly to this:
if __name__ == '__main__':
configure_logging() # whatever configuration you need to do
main()
This allows REPL usage without logging being printed (other than WARNING or above), while logging occurs if the application is run.
What happens if you remove addHandler and setLevel from bar.py ? I don't see why you would want your package to actually do the logging instead of generating log messages, and the NullHandler sounds like it would be dropping your messages alltogether.
I have a question about python standard logging mechanism. So if I use logging.config.fileConfig to load my configuration file, then I create loggers for some modules using logging.getLogger test them right after creation and they work. Now if I call logging.config.fileConfig again with the same configuration file and create loggers for some other module would the previos ones still work ? Basically for the following logic:
logging.config.fileConfig(config_file)
logger1 = logging.getLogger(module1)
logger2 = logging.getLogger(module2)
logging.config.fileConfig(config_file)
logger3 = logging.getLogger(module3)
config_file is the same in both calls. Should logger1 and logger2 be functional ? How about if config_file is different in those calls? Currently my logger1 and logger2 are not working after i load a new config_file. So first step is to check if this is normal behaviour. If so is it possible to make this work without merging the two config_files into one big one?
Regards,
Bogdan
Config files are intended to completely replace the existing configuration with whatever is in the confguration - any loggers which are not named in the configuration, or children thereof, are disabled by fileConfig(), as documented here. you can prevent this disabling, but only on recent Python versions. It's not generally good practice to call fileConfig() multiple times in a program, unless you have a specific need to do so. It's not forbidden, but it's not usual.
A common usage involves configuring handlers on the root logger and perhaps one or two top-level loggers; does this apply to you?