I have a logger setup like this:
import logging
from logging.handlers import RotatingFileHandler
import sys
# root logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
# create a file logger
handler = RotatingFileHandler('log/core-application.log', maxBytes=1024*1024*1, backupCount=3)
handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(funcName)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
# create stdout logger
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(funcName)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info('Logging all setup')
On my development system this logging entry is working ok. It logs what you see here:
logger.info('message key: {}'.format('2018-10-19_00:20:56_\xd8\xa7\xd9\x84\xd8\xb5\xd9\x88\xd8\xa7\xd9\x81/'))
The same line on my ubuntu AWS EC2 server gives me an error:
--- Logging error --- Traceback (most recent call last): File "/usr/lib/python3.5/logging/init.py", line 982, in emit
stream.write(msg) UnicodeEncodeError: 'ascii' codec can't encode characters in position 108-113: ordinal not in range(128)
Can anybody image why this is?
btw:
EC2 server locales are:
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8
Thanks in advance.
What worked for me is adding encoding='utf8' arg to the handler :
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
log_path = '/var/log/app'
log_file = '%s/%s.log' % (log_path, name)
create_log_file(log_file)
handler = logging.FileHandler(log_file, encoding='utf8')
handler.setFormatter(formatter)
logger.addHandler(handler)
I just found the solution and would like to share the answer:
It was connected to supervisor controlling the python app on the server.
Somehow supervisor config needs to contain the environment variables below to be able to print UTF-8 to stdout.
[supervisord]
environment=LC_ALL='en_US.UTF-8',LANG='en_US.UTF-8'
edit:
https://docs.python.org/3.7/whatsnew/3.7.html#whatsnew37-pep538
Python 3.7 seems to be designed to automatically solve this issue.
After having searched for a long moment, this solution worked for me under Windows 10 x64 with Python 3.7.6, whenever a `logger.error('message', exc_info=True) would fail because the traceback contained invalid characters.
Adding a u before the formatter string to force an unicode message:
#formatter = logging.Formatter('%(asctime)s - %(funcName)s - %(levelname)s - %(message)s'
formatter = logging.Formatter(u'%(asctime)s - %(funcName)s - %(levelname)s - %(message)s'
Hope it helps whoever may read this.
Related
I'm using the logging library and would like to log on to the console and store those logs in a file.
logger = logging.getLogger(module)
handler = logging.StreamHandler()
formatter = logging.Formatter(
'%(asctime)s [%(name)s] %(levelname)-8s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
filehandler=logging.FileHandler('logfile.log')
logger.addHandler(filehandler)
logger.info('hello')
my goal is to see logs in the below format.
[time] [module] [info/error] [log-msg]
I'm getting this format in the console but in log file I'm getting only log-msg
filehandler.setFormatter(formatter)
I have a bootstrap script for a Raspberry Pi that runs in python. I am looking to create a logger that logs to a file as well as to the console.
I was going to do something like this:
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(threadName)-12.12s] [%(levelname)-5.5s] %(message)s",
handlers=[
logging.FileHandler("{0}/{1}.log".format(logPath, fileName)),
logging.StreamHandler()
])
But what I would really like is to log INFO to the StreamHandler and DEBUG to the FileHandler... I cannot seem to figure that out.
Can anyone help me out?
Using Python 3.7.5
You could build the logger yourself (either through a config file or in pure python)
The tricky thing that I have wasted several hours on is forgetting to set the log level on the logger as well as on each of the handlers. Ensure that the logger is as permissive as the most permissive handler.
example script
# emits the info line to the console and
# both the info & debug lines to the log file
# test_pylog.py
import logging
log_format = logging.Formatter(
'%(asctime)s %(threadName)s %(levelname)s %(message)s'
)
logger = logging.getLogger(__name__)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
console_handler.setFormatter(log_format)
logger.addHandler(console_handler)
file_handler = logging.FileHandler('logfile.txt')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(log_format)
logger.addHandler(file_handler)
logger.setLevel(logging.DEBUG)
if __name__ == '__main__':
logger.debug('Panic! at the disco')
logger.info('Weezer')
I am using below configuration for logging in python,
logger = logging.getLogger(__name__)
hdlr = logging.FileHandler(r'/var/log/resource_checker.log')
formatter = logging.Formatter('%(asctime)s : %(name)s : %(levelname)s : %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.INFO)
This works well for me. However if I add logging.basicConfig(level=logging.DEBUG) with existing logging configuration, all my logs pops up on screen screen (with additional DEBUG logs, which is good to have for me) and logfile does not populate at all.
how can i push those logs back to log file as my older config.?
This will log everything to a file, all from the logging.basicConfig:
import logging
logging.basicConfig(
level=logging.DEBUG,
filename='/var/log/resource_checker.log',
format='%(asctime)s : %(name)s : %(levelname)s : %(message)s'
)
logger = logging.getLogger(__name__)
I'm trying to figure out how to capture messages generated by Python/NumPy intractive shell when running my script. I would like to log all generated by console messages (errors, warnings) to same file as defined in my code log messages with time stamps:
def LogToFile():
global logger
logger = logging.getLogger('MyApp')
logger.setLevel(logging.DEBUG)
file_log_handler = RotatingFileHandler('logfile.log', maxBytes=1024, backupCount=5)
logger.addHandler(file_log_handler)
stderr_log_handler = logging.StreamHandler()
logger.addHandler(stderr_log_handler)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_log_handler.setFormatter(formatter)
stderr_log_handler.setFormatter(formatter)
return logger
Afaik, you'd have to specify this in basicConfig, not in your logger:
logging.basicConfig(filename=LOG_FILE,
level=logging.DEBUG)
before you do
logger = logging.getLogger('MyApp')
logger.setLevel(logging.DEBUG)
I am trying to get module level logging via code working for three outputs: file, console and application internal(QTextEdit).
I can get all three loggers working with the code below but the application internal logger is not logging all events and the console logger (only) prints each line twice.
I have tried using
logging.getLogger(__name__)
for the file logger instead of root (no logs generated), same for the console (works fine with only 1 line per log output) and same for the MyLogHandler (no logs generated) and tried various combinations of root logger and 'name' logger but can't get all logs working and console only printing one line per log event.
def configCodeRootExample_(self):
logFileName = self.getLogLocation()
rootLogger = logging.getLogger('')
#This logger works
fileLogger = logging.FileHandler(logFileName)
fileLogger.setLevel(logging.INFO)
fileFormatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
fileLogger.setFormatter(fileFormatter)
rootLogger.addHandler(fileLogger)
#This logger works but prints output twice
consoleFormatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s - %(module)s - %(funcName)s - %(lineno)d - %(message)s')
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
console.setFormatter(consoleFormatter)
rootLogger.addHandler(console)
#This logger works but only logs a subset of DEBUG events and no INFO events
myLogHandler = GSLLogHandler()
myLogHandler.setLevel(logging.DEBUG)
myLogHandler.setFormatter(fileFormatter)
rootLogger.addHandler(myLogHandler)
also for the record here is the log handler to output to a listening QTextEdit:
import logging
from loggerpackage.logsignals import LogSignals
class MyLogHandler(logging.Handler):
def __init__(self):
logging.Handler.__init__(self)
self.logSignals = LogSignals()
def emit(self, logMsg):
logMsg = self.format(logMsg)
self.logSignals.logEventTriggered.emit(logMsg)
If I change the console logger to the module level:
logger = logging.getLogger(__name__)
consoleFormatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s - %(module)s - %(funcName)s - %(lineno)d - %(message)s')
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
console.setFormatter(consoleFormatter)
logger.addHandler(console)
Then only one line is printed for each log event but the formatting is incorrect, it seems to be some sort of default formatter
See here for a solution to the duplicate console logging: How to I disable and re-enable console logging in Python?
logger = logging.getLogger()
lhStdout = logger.handlers[0]
... add log handlers
logger.removeHandler(lhStdout)
The issue I was having with the MyLogHandler was that the slot on the QTextEdit wasn't connected in time to receive the first few DEBUG and INFO events.