I'm struggling to get Python to log the way I want. I've added some context to my log statements by adding a filter and updating the format string to print what's in the filter. That works as expected as long as all the code is mine.
But if a 3rd party module logs something, it throws an error because that module doesn't know about my filter.
How do I get context into my logs without blowing up 3rd party module logging?
This code works fine in my modules. But if a 3rd party module wants to log something, they don't know about my ContextFilter, which details the nstid I want in my log messages.
import logging
import sys
import boto3
from ContextFilter import ContextFilter
logging.basicConfig(
format='%(asctime)s %(levelname)-8s nstid:%(nstid)8s %(message)s',
handlers=[logging.StreamHandler(sys.stdout)],
level=logging.INFO,
datefmt='%Y-%m-%d %H:%M:%S'
)
log = logging.getLogger(__name__)
log.addFilter(ContextFilter())
log.info("important information")
I was able to get what I needed using a CustomAdapter instead of a ContextFilter, but I'm still interested in other solutions:
CustomAdapter.py:
import logging
import os
class CustomAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
nstid = os.environ['NSTID'] if 'NSTID' in os.environ else None
return '[NSTID: %s] %s' % (nstid, msg), kwargs
import logging
import os
import sys
import boto3
from CustomAdapter import CustomAdapter
logging.basicConfig(
format='%(asctime)s %(levelname)-8s %(message)s',
handlers=[logging.StreamHandler(sys.stdout)],
level=logging.INFO,
datefmt='%Y-%m-%d %H:%M:%S'
)
log = CustomAdapter(logging.getLogger(__name__), {})
log.info("important information")
sqs = boto3.resource('sqs', region_name=os.environ['AWS_REGION'])
outputs:
2020-04-13 13:24:38 INFO [NSTID: 24533] important information
2020-04-13 13:24:38 INFO Found credentials in shared credentials file: ~/.aws/credentials
the first line is from my code, the second from boto3
Related
I have a code structure that looks like this:
.
├── scripts
│ └── test.py
└── utils
├── __init__.py
└── lib.py
In my utils/__init__.py file I've set up logging like so:
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
ch = logging.FileHandler("logfile", mode="w")
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter(
"%(levelname)s:%(name)s:[%(funcName)s]:%(message)s", "%m/%d/%Y %I:%M:%S %p"
)
ch.setFormatter(formatter)
logger.addHandler(ch)
This works, and if I log in lib.py using code like so
LOGGER = logging.getLogger(__name__)
def a_function():
LOGGER.info('This works')
then everything runs and gets logged.
However, I often find myself wanting to run different scripts while in the same environment and I'd like to have a log that's named differently for each run (I don't want them writing over each other or appending to the same log). What is the best way to modify the logging name when logging is set up like this? I've tried doing things like the following in scripts/test.py:
import logging
from utils.lib import a_function
def main():
# rename handler
fh = logging.FileHandler("logfile2", "w")
formatter = logging.Formatter(
"%(levelname)s:%(name)s:[%(funcName)s]:%(message)s", "%m/%d/%Y %I:%M:%S %p"
)
fh.setFormatter(formatter)
log = logging.getLogger()
log.handlers.clear()
log.addHandler(fh)
a_function()
if __name__ == '__main__':
main()
This works in adding a new log called logfile2, but I still get logging in the original logfile. I would have thought that the log.handlers.clear() command would have cleared that out.
What am I doing wrong here?
This is my script. I have edited this script for log filethrough some information on internet. Still i am unable to reveive log files for error. Anyone can help me solve this. This is my script.
from pyspark.sql.functions import udf
from datetime import datetime
from math import floor
from pyspark.context import SparkContext
from pyspark.sql import DataFrame
from pyspark.sql.functions import expr`
import os
import sys
import logging
import logging.handlers
log = logging.getLogger('log_file')
handler = logging.FileHandler("spam.log")
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
log.addHandler(handler)
sys.stderr.write = log.error
sys.stdout.write = log.info
sc = SparkContext('local')
spark = SparkSession(sc)
sc.addPyFile("udf3.py")
from udf3 import BW_diag
bw = udf(BW_diag);`
My pysparkshell previously used to show me the errors in shell itself but now after this log script i haved copied from somewhere, I am not getting errors on shell. I also want log file because i have to run my scrpit in oozie and have to get error logs for checking my script.
More simply - first i have to run script for non restricted in Pyspark sample data and again place in OOzie for main data run.
Thank you experts!
When I run pytest, I am not seeing any log messages, how can I fix it, I tried to search for pytest.ini file which is not present in my local, I am new to pytest and need some help.
import test_todo_lib
import logging
logging.basicConfig(format='%(asctime)s - %(message)s', level=logging.CRITICAL)
# NOTE: Run pytest using --capture=tee-sys option inorder to display standard output
def test_basic_unit_tests(browser):
test_page = test_todo_lib.ToDoAppPage(browser)
test_page.load()
logging.info("Check the basic functions")
I want to log some python code which loads in a trained keras model. For some reason, (python) logging doesn't not work if the keras load_model is imported. But (python) logging works fine if I don't import keras.
This works:
import logging
LOG_FORMAT = "%(levelname)s %(asctime)s - %(message)s"
logging.basicConfig(filename="logs/my_logs.log",
level=logging.DEBUG,
format=LOG_FORMAT)
logger = logging.getLogger()
def do_stuff():
logging.info("About to do stuff")
... stuff gets done here...
This doesn't work
import logging
from keras.models import load_model
my_model = load_model("fetch_preprocess/lstm_model.h5")
LOG_FORMAT = "%(levelname)s %(asctime)s - %(message)s"
logging.basicConfig(filename="logs/my_logs.log",
level=logging.DEBUG,
format=LOG_FORMAT)
logger = logging.getLogger()
def do_stuff():
logging.info("About to do stuff")
pred = my_model.predict(data)
... stuff gets done here...
By "doesn't work", I mean the logging module doesn't
create a new log file
write to that file anytime logging is called
but no error's are ever thrown. So I find this strange.
I believe Keras / tensorflow is interfering with the logging (as the only change I make to the code is to exclude Keras and then logging works fine).
Is there some way to suppress whatever keras is doing in the background so I can use the python logging?
This issue is because of the bug in the version 0.7.0 of the package, abseil and it has been fixed in its 0.8.0 version, as mentioned in this Github link.
The solution to this problem is to upgrade the version of the package abseil to greater than or equal to 0.8.
The sample code that
creates a new log file and
writes to that file anytime logging is called
is shown below:
`
!pip install --upgrade absl-py
import logging
from keras.models import load_model
my_model = load_model("fetch_preprocess/lstm_model.h5")
LOG_FORMAT = "%(levelname)s %(asctime)s - %(message)s"
logging.basicConfig(filename="logs/my_logs.log",
level=logging.DEBUG,
format=LOG_FORMAT)
logger = logging.getLogger()
def do_stuff():
logging.info("About to do stuff")
pred = my_model.predict(data)
... stuff gets done here...
do_stuff()
`
Hope this helps. Happy Learning!
As you mentioned, I also think that Keras is simply replacing your logger with its own. You can, however, do the following:
import logging as my_logging
from keras.models import load_model
my_model = load_model("fetch_preprocess/lstm_model.h5")
LOG_FORMAT = "%(levelname)s %(asctime)s - %(message)s"
my_logging.basicConfig(filename="logs/my_logs.log",
level=my_logging.DEBUG,
format=LOG_FORMAT)
my_logger = my_logging.getLogger()
Write your code with a nice logger
import logging
def init_logging():
logFormatter = logging.Formatter("[%(asctime)s] %(levelname)s::%(module)s::%(funcName)s() %(message)s")
rootLogger = logging.getLogger()
LOG_DIR = os.getcwd() + '/' + 'logs'
if not os.path.exists(LOG_DIR):
os.makedirs(LOG_DIR)
fileHandler = logging.FileHandler("{0}/{1}.log".format(LOG_DIR, "g2"))
fileHandler.setFormatter(logFormatter)
rootLogger.addHandler(fileHandler)
rootLogger.setLevel(logging.DEBUG)
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
rootLogger.addHandler(consoleHandler)
return rootLogger
logger = init_logging()
works as expected. Logging using logger.debug("Hello! :)") logs to file and console.
In a second step you want to import an external module which is also logging using logging module:
Install it using pip3 install pymisp (or any other external module)
Import it using from pymisp import PyMISP (or any other external module)
Create an object of it using self.pymisp = PyMISP(self.ds_model.api_url, self.ds_model.api_key, False, 'json') (or any other...)
What now happens is, that every debug log output from the imported module is getting logged to the log file and the console. The question now is, how to set a different (higher) log level for the imported module.
As Meet Sinoja and anishtain4 pointed out in the comments, the best and most generic method is to retrieve the logger by the name of the imported module as follows:
import logging
import some_module_with_logging
logging.getLogger("some_module_with_logging").setLevel(logging.WARNING)
Another option (though not recommended if the generic method above works) is to extract the module's logger variable and customize it to your needs. Most third-party modules store it in a module-level variable called logger or _log. In your case:
import logging
import pymisp
pymisp.logger.setLevel(logging.INFO)
# code of module goes here
A colleague of mine helped with this question:
Get a named logger yourLogger = logging.getLogger('your_logger')
Add a filter to each handler prevents them to print/save other logs than yours
for handler in logging.root.handlers:
handler.addFilter(logging.Filter('your_logger'))