Develop Reference

Prevent Generation of Log File with Python logging

I have a simple script that I run as an exe file on Windows. However, when I am developing the script, I run it from the command line and use the logging module to output debug info to a log file. I would like to turn off the generation of the log file for my production code. How would I go about doing that?
This is the logging config I have setup now:
import logging
...
logging.basicConfig(filename='file.log',
filemode="w",
level=logging.DEBUG,
format="%(asctime)s: %(name)s - %(levelname)s - %(message)s",
datefmt='%d-%b-%y %H:%M:%S',
)
...
logging.debug("Debug message")

If you don't mind the generation of an empty log file for production, you can simply increase the threshold of logging to a level above logging.DEBUG, such as logging.INFO, so that messages logged with logging.debug won't get output to the log file:
logging.basicConfig(filename='file.log', # creates an empty file.log
filemode="w",
level=logging.INFO,
format="%(asctime)s: %(name)s - %(levelname)s - %(message)s",
datefmt='%d-%b-%y %H:%M:%S',
)
logging.debug("Debug message") # nothing would happen
logging.info("FYI") # logs 'FYI'
If you don't want logging to function at all, an easy approach is to override logging with a Mock object:
import logging
from unittest.mock import Mock
environment = 'production'
if environment == 'production':
logging = Mock()
...
logging.basicConfig(...) # nothing would happen
logging.debug(...) # nothing would happen

python3 logger - UnicodeEncodeError

I have a logger setup like this:
import logging
from logging.handlers import RotatingFileHandler
import sys
# root logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
# create a file logger
handler = RotatingFileHandler('log/core-application.log', maxBytes=1024*1024*1, backupCount=3)
handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(funcName)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
# create stdout logger
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(funcName)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info('Logging all setup')
On my development system this logging entry is working ok. It logs what you see here:
logger.info('message key: {}'.format('2018-10-19_00:20:56_\xd8\xa7\xd9\x84\xd8\xb5\xd9\x88\xd8\xa7\xd9\x81/'))
The same line on my ubuntu AWS EC2 server gives me an error:
--- Logging error --- Traceback (most recent call last): File "/usr/lib/python3.5/logging/init.py", line 982, in emit
stream.write(msg) UnicodeEncodeError: 'ascii' codec can't encode characters in position 108-113: ordinal not in range(128)
Can anybody image why this is?
btw:
EC2 server locales are:
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8
Thanks in advance.

What worked for me is adding encoding='utf8' arg to the handler :
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
log_path = '/var/log/app'
log_file = '%s/%s.log' % (log_path, name)
create_log_file(log_file)
handler = logging.FileHandler(log_file, encoding='utf8')
handler.setFormatter(formatter)
logger.addHandler(handler)

I just found the solution and would like to share the answer:
It was connected to supervisor controlling the python app on the server.
Somehow supervisor config needs to contain the environment variables below to be able to print UTF-8 to stdout.
[supervisord]
environment=LC_ALL='en_US.UTF-8',LANG='en_US.UTF-8'
edit:
https://docs.python.org/3.7/whatsnew/3.7.html#whatsnew37-pep538
Python 3.7 seems to be designed to automatically solve this issue.

After having searched for a long moment, this solution worked for me under Windows 10 x64 with Python 3.7.6, whenever a `logger.error('message', exc_info=True) would fail because the traceback contained invalid characters.
Adding a u before the formatter string to force an unicode message:
#formatter = logging.Formatter('%(asctime)s - %(funcName)s - %(levelname)s - %(message)s'
formatter = logging.Formatter(u'%(asctime)s - %(funcName)s - %(levelname)s - %(message)s'
Hope it helps whoever may read this.

python log pops up on console

I am using below configuration for logging in python,
logger = logging.getLogger(__name__)
hdlr = logging.FileHandler(r'/var/log/resource_checker.log')
formatter = logging.Formatter('%(asctime)s : %(name)s : %(levelname)s : %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.INFO)
This works well for me. However if I add logging.basicConfig(level=logging.DEBUG) with existing logging configuration, all my logs pops up on screen screen (with additional DEBUG logs, which is good to have for me) and logfile does not populate at all.
how can i push those logs back to log file as my older config.?

This will log everything to a file, all from the logging.basicConfig:
import logging
logging.basicConfig(
level=logging.DEBUG,
filename='/var/log/resource_checker.log',
format='%(asctime)s : %(name)s : %(levelname)s : %(message)s'
)
logger = logging.getLogger(__name__)

flask + uwsgi: No handlers could be found for logger

I am running a python flask app using uwsgi and nginx. I am having trouble getting the app modules to log. If i run the flask app by itself, i can see the logs properly formatted, but in uwsgi, i see 'no handlers could be found for logger...' and the logs are missing. prints show up fine. Could someone help with what I am doing wrong?
Thanks
I run uwsgi as
/usr/local/bin/uwsgi --ini /root/uwsgi.ini
# cat /root/uwsgi.ini
[uwsgi]
base=/root/mainapp
app = mainapp
module = %(app)
pythonpath = %(base)
socket = /tmp/mainapp.sock
chmod-socket = 666
callable = app
logto = /var/log/mainapp/app.log
paste-logger = %p
[formatters]
keys: detailed
[handlers]
keys: console
[loggers]
keys: root, module1, module2, module3
[formatter_detailed]
format: %(asctime)s %(name)s:%(levelname)s %(module)s:%(lineno)d: %(message)s
[handler_console]
class: StreamHandler
args: []
formatter: detailed
[logger_root]
level: DEBUG
handlers:
[logger_module1]
level: DEBUG
qualname: module1
handlers: console
[logger_module2]
level: DEBUG
qualname: module2
handlers: console
[logger_module3]
level: DEBUG
qualname: module3
handlers: console
and in the module, i call
import logging
log = logging.getLogger('module1')
log.info('hello world')

Have you configured Flask to use the logger? Off the top of my head something like this should work
app.config['LOG_FILE'] = 'application.log'
# Configure logger.
if not app.debug:
import logging
from logging import FileHandler
file_handler = FileHandler(app.config['LOG_FILE'])
file_handler.setLevel(logging.WARNING)
app.logger.addHandler(file_handler)

looks like i have to instantiate flask's app.logger before i can do anything...this did the trick...set addhandler on app.logger and
import logging
import logging.config
shandler = logging.StreamHandler()
shandler.setLevel(logging.DEBUG)
app.logger.addHandler(shandler)
logging.config.fileConfig('logging.conf')

Python module level logging configuration via code issue

I am trying to get module level logging via code working for three outputs: file, console and application internal(QTextEdit).
I can get all three loggers working with the code below but the application internal logger is not logging all events and the console logger (only) prints each line twice.
I have tried using
logging.getLogger(__name__)
for the file logger instead of root (no logs generated), same for the console (works fine with only 1 line per log output) and same for the MyLogHandler (no logs generated) and tried various combinations of root logger and 'name' logger but can't get all logs working and console only printing one line per log event.
def configCodeRootExample_(self):
logFileName = self.getLogLocation()
rootLogger = logging.getLogger('')
#This logger works
fileLogger = logging.FileHandler(logFileName)
fileLogger.setLevel(logging.INFO)
fileFormatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
fileLogger.setFormatter(fileFormatter)
rootLogger.addHandler(fileLogger)
#This logger works but prints output twice
consoleFormatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s - %(module)s - %(funcName)s - %(lineno)d - %(message)s')
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
console.setFormatter(consoleFormatter)
rootLogger.addHandler(console)
#This logger works but only logs a subset of DEBUG events and no INFO events
myLogHandler = GSLLogHandler()
myLogHandler.setLevel(logging.DEBUG)
myLogHandler.setFormatter(fileFormatter)
rootLogger.addHandler(myLogHandler)
also for the record here is the log handler to output to a listening QTextEdit:
import logging
from loggerpackage.logsignals import LogSignals
class MyLogHandler(logging.Handler):
def __init__(self):
logging.Handler.__init__(self)
self.logSignals = LogSignals()
def emit(self, logMsg):
logMsg = self.format(logMsg)
self.logSignals.logEventTriggered.emit(logMsg)
If I change the console logger to the module level:
logger = logging.getLogger(__name__)
consoleFormatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s - %(module)s - %(funcName)s - %(lineno)d - %(message)s')
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
console.setFormatter(consoleFormatter)
logger.addHandler(console)
Then only one line is printed for each log event but the formatting is incorrect, it seems to be some sort of default formatter

See here for a solution to the duplicate console logging: How to I disable and re-enable console logging in Python?
logger = logging.getLogger()
lhStdout = logger.handlers[0]
... add log handlers
logger.removeHandler(lhStdout)
The issue I was having with the MyLogHandler was that the slot on the QTextEdit wasn't connected in time to receive the first few DEBUG and INFO events.