Write your code with a nice logger
import logging
def init_logging():
logFormatter = logging.Formatter("[%(asctime)s] %(levelname)s::%(module)s::%(funcName)s() %(message)s")
rootLogger = logging.getLogger()
LOG_DIR = os.getcwd() + '/' + 'logs'
if not os.path.exists(LOG_DIR):
os.makedirs(LOG_DIR)
fileHandler = logging.FileHandler("{0}/{1}.log".format(LOG_DIR, "g2"))
fileHandler.setFormatter(logFormatter)
rootLogger.addHandler(fileHandler)
rootLogger.setLevel(logging.DEBUG)
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
rootLogger.addHandler(consoleHandler)
return rootLogger
logger = init_logging()
works as expected. Logging using logger.debug("Hello! :)") logs to file and console.
In a second step you want to import an external module which is also logging using logging module:
Install it using pip3 install pymisp (or any other external module)
Import it using from pymisp import PyMISP (or any other external module)
Create an object of it using self.pymisp = PyMISP(self.ds_model.api_url, self.ds_model.api_key, False, 'json') (or any other...)
What now happens is, that every debug log output from the imported module is getting logged to the log file and the console. The question now is, how to set a different (higher) log level for the imported module.
As Meet Sinoja and anishtain4 pointed out in the comments, the best and most generic method is to retrieve the logger by the name of the imported module as follows:
import logging
import some_module_with_logging
logging.getLogger("some_module_with_logging").setLevel(logging.WARNING)
Another option (though not recommended if the generic method above works) is to extract the module's logger variable and customize it to your needs. Most third-party modules store it in a module-level variable called logger or _log. In your case:
import logging
import pymisp
pymisp.logger.setLevel(logging.INFO)
# code of module goes here
A colleague of mine helped with this question:
Get a named logger yourLogger = logging.getLogger('your_logger')
Add a filter to each handler prevents them to print/save other logs than yours
for handler in logging.root.handlers:
handler.addFilter(logging.Filter('your_logger'))
Related
When I run pytest, I am not seeing any log messages, how can I fix it, I tried to search for pytest.ini file which is not present in my local, I am new to pytest and need some help.
import test_todo_lib
import logging
logging.basicConfig(format='%(asctime)s - %(message)s', level=logging.CRITICAL)
# NOTE: Run pytest using --capture=tee-sys option inorder to display standard output
def test_basic_unit_tests(browser):
test_page = test_todo_lib.ToDoAppPage(browser)
test_page.load()
logging.info("Check the basic functions")
I'm struggling to get Python to log the way I want. I've added some context to my log statements by adding a filter and updating the format string to print what's in the filter. That works as expected as long as all the code is mine.
But if a 3rd party module logs something, it throws an error because that module doesn't know about my filter.
How do I get context into my logs without blowing up 3rd party module logging?
This code works fine in my modules. But if a 3rd party module wants to log something, they don't know about my ContextFilter, which details the nstid I want in my log messages.
import logging
import sys
import boto3
from ContextFilter import ContextFilter
logging.basicConfig(
format='%(asctime)s %(levelname)-8s nstid:%(nstid)8s %(message)s',
handlers=[logging.StreamHandler(sys.stdout)],
level=logging.INFO,
datefmt='%Y-%m-%d %H:%M:%S'
)
log = logging.getLogger(__name__)
log.addFilter(ContextFilter())
log.info("important information")
I was able to get what I needed using a CustomAdapter instead of a ContextFilter, but I'm still interested in other solutions:
CustomAdapter.py:
import logging
import os
class CustomAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
nstid = os.environ['NSTID'] if 'NSTID' in os.environ else None
return '[NSTID: %s] %s' % (nstid, msg), kwargs
import logging
import os
import sys
import boto3
from CustomAdapter import CustomAdapter
logging.basicConfig(
format='%(asctime)s %(levelname)-8s %(message)s',
handlers=[logging.StreamHandler(sys.stdout)],
level=logging.INFO,
datefmt='%Y-%m-%d %H:%M:%S'
)
log = CustomAdapter(logging.getLogger(__name__), {})
log.info("important information")
sqs = boto3.resource('sqs', region_name=os.environ['AWS_REGION'])
outputs:
2020-04-13 13:24:38 INFO [NSTID: 24533] important information
2020-04-13 13:24:38 INFO Found credentials in shared credentials file: ~/.aws/credentials
the first line is from my code, the second from boto3
I just started Python and I am struggling to use Logger. I have two python files: app.py and liba.py. I want to setup logging at app.py and use it for liba.py (and another libraries). Do you have any good ideas or can you share any references?
file structure
entry.py
lib/liba.py
app.py
#! /usr/bin/env python3
import logging
logger = logging.getLogger(__name__)
from lib import liba
handler = logging.FileHandler('/tmp/app.log', 'a+')
logger.addHandler(handler)
logger.warn('sample')
liba.out()
lib/liba.py
#! /usr/bin/env python3
import logging
logger = logging.getLogger(__name__)
def out():
logger.warn('liba')
run python
$ python3 app.py
liba
app.py output log to the logfile. liba.py does not output the log into the file. I want to save logs in the same file.
Do like so:
app.py
#! /usr/bin/env python3
import logging
logger = logging.getLogger()
handler = logging.FileHandler('/tmp/app.log', 'a+')
logger.addHandler(handler)
logger.warn('sample')
from lib import liba
liba.out()
lib/liba.py
#! /usr/bin/env python3
import logging
def out():
logging.warn('liba')
You don't need to instantiate the logging, unless you want to configure handlers, which you only do in your main script. Then all logging will go to the root logger which is what you get when instantiating with no specific name logging.getLogger(). I like to use it this way as you don't need to match names across all your modules for it to work. In your modules you just send log messages out by using logging.warn('blabla'). You further need to make sure you define all your handlers before any call to logging.warn is made, otherwise some default handler will take its place.
I used to Python logging, it works fine. The logging.basicConfig(...) set in one module (a some.py file), then we can use logging every where. Obviously, logging is global.
The question is how logging find it's settings, when we not call the module where basicConfig(...) appeared (in some.py file )? Is logging scan all the packages?
Even the logging.basicConfig(...) put into an any.py and the module (any.py) never get imported, or not used anywhere, the logging setting take effect!
To understand logging you have dive into Python's standard library sources.
Here is the trick:
#/usr/lib/python3.2/logging/__init__.py
...
root = RootLogger(WARNING)
Logger.root = root
Logger.manager = Manager(Logger.root)
...
# and
def basicConfig(**kwargs):
...
hdlr = StreamHandler(stream)
fs = kwargs.get("format", BASIC_FORMAT)
dfs = kwargs.get("datefmt", None)
style = kwargs.get("style", '%')
fmt = Formatter(fs, dfs, style)
hdlr.setFormatter(fmt)
root.addHandler(hdlr)
So, when you call basicconfig() with certain parameters, root logger is set.
Finally getLogger:
def getLogger(name=None):
"""
Return a logger with the specified name, creating it if necessary.
If no name is specified, return the root logger.
"""
if name:
return Logger.manager.getLogger(name)
else:
return root
I think there is no magic scanning here.
Try to test it this way in a separate test directory:
test/main.py:
import logging
logging.info('test')
test/any.py:
import logging
logging.basicConfig(filename='test.log', level=logging.INFO)
python main.py
Result: NO test.log file.
Now let's update the test:
test/main.py:
import logging
import any
logging.info('test')
python main.py
Result: new test.log file with INFO:root:test string inside.
So I guess that any.py in your case is imported somehow,
despite your expectations.
You may find the way any.py is imported easily,
just add few lines there:
test/any.py:
from traceback import print_stack
print_stack()
...
python main.py
Result:
File "main.py", line 2, in
import any
File "any.py", line 2, in
print_stack()
This stack shows that any.py is imported from main.py.
I hope you will find where it is imported from in your case.
I have a Python program that consists of several modules. The "main" module creates a file variable log_file for logging the output; all the other modules would need to write to that file as well.
However, I don't want to import the "main" module into other modules, since it would be a really weird dependency (not to mention it might not even work due to circular dependency).
Where, then, should I store the log_file variable?
EDIT:
Following #pyfunc answer - would this be ok:
--- config.py ---
# does not mention log_file
# unless it's required for syntax reasons; in which case log_file = None
# ...
--- main.py ---
from datetime import datetime
import config.py
log_filename = str(datetime.now()) + '.txt'
config.log_file = open(log_filename, 'w')
# ...
--- another_module.py ---
import config.py
# ...
config.log_file.write(some_stuff)
Put the "global" variable in a "settings" module.
settings.py
log_file = "/path/to/file"
main.py
import settings
import logging
logging.basicConfig(filename=settings.log_file,level=logging.DEBUG)
logging.debug("This should go to the log file")
other_module.py
import logging
logging.debug("This is a message from another place.")
While the logging module may solve your immediate problem and many others, the settings module pattern is useful for a lot of other things besides log file names. It is used by Django to configure just about everything.
One way is to take all that code into another module so that you could import it in main file and other modules as well.
Hope you have checked on : http://docs.python.org/library/logging.html